Stanford CS229 I Machine Learning I Building Large Language Models (LLMs)

TL;DR

• Yann Dubois (Stanford PhD student, co-author of Alpaca and AlpacaFarm) gives a 1h44m guest lecture in CS229 (Summer 2024, 2.8M views) on building a ChatGPT-style model. He splits the work into five parts (architecture, loss, data, evaluation, systems) and skips architecture on purpose: academia works on it, but "what matters in practice is mostly the three other topics".
• Pre-training is next-token prediction with cross-entropy loss, which is the same as maximizing the text's log-likelihood. Tokenizers (byte-pair encoding, about 3-4 letters per token) exist because words break on typos and characters make sequences too long for attention's quadratic cost. Data is the hard part. Common Crawl (~250B pages, ~1 PB) goes through text extraction, safety and PII filters, deduplication, heuristic and model-based quality filters, and domain reweighting, ending at around 15T tokens.
• Scaling laws decide the budget. Loss falls predictably on a log-log plot as compute, data and parameters grow, with no plateau yet. Chinchilla's optimum is 20 tokens per parameter; production models train on ~150 per parameter because smaller models are cheaper to serve. Llama 3 405B: 15.6T tokens, 3.8e25 FLOPs (6 × params × tokens), ~70 days on 16,000 H100s, about $52M of compute plus ~$25M in salaries.
• In post-training, SFT on a few thousand examples barely benefits from more data (LIMA: 2k vs 32k), because it only selects which kind of internet writer to imitate. RLHF then optimizes human preferences, first with a reward model plus PPO (the original ChatGPT recipe) and now mostly with DPO, which reaches the same optimum using plain maximum likelihood. LLM judges replace human labelers at ~50x lower cost, with the length bias both share.
• On systems, GPUs are bottlenecked on memory and communication more than math. A model FLOP utilization of 50% counts as good; mixed precision (16-bit compute, 32-bit weights) and fused kernels (torch.compile, about 2x) are the first tricks.

Key moments

• [00:56] Five components — architecture, training loss, data, evaluation, systems; he drops architecture because industry's effort goes into the other three.
• [03:42] Language modeling — a probability distribution over token sequences: "the mouse ate the cheese" beats a garbled version (syntax) and "the cheese ate the mouse" (semantics) (frame_0002). Autoregressive models factor it with the chain rule.
• [10:45] Tokenizers — byte-pair encoding merges the most frequent pairs; numbers like "327" become a single token, which hurts arithmetic; GPT-4 changed how code indentation tokenizes.
• [19:10] Evaluation — perplexity (2^average loss, between 1 and vocabulary size) fell from ~70 to under 10 between 2017 and 2023; academic benchmarks moved to HELM, the Hugging Face leaderboard and MMLU (frame_0004). The same Llama 65B scored 63.7 or 48.8 on MMLU depending on how it was evaluated.
• [28:36] Data pipeline — a random Common Crawl page is an unfinished sentence in HTML; Wikipedia-referenced pages train the quality classifier; code gets upweighted "because it helps reasoning"; training ends on high-quality data at a low learning rate (frame_0005).
• [40:55] Scaling laws — "overfitting doesn't happen with large language models"; tune hyperparameters on small models, fit a curve, then spend 27 of 30 days on the big run. Transformers beat LSTMs on both slope and intercept.
• [49:27] Chinchilla — iso-FLOP curves give 20 tokens per parameter; inference cost pushes practice toward ~150.
• [55:16] Llama 3 405B, back of the envelope — 3.8e25 FLOPs, about 2x under the 1e26 threshold of the US executive order; 26M GPU-hours (Meta reported 30M); ~$75M total; ~4,000 t CO2e, about 2,000 JFK-London round trips (frame_0007).
• [59:56] Post-training — raw GPT-3 answers "explain the moon landing to a six-year-old" with more questions; SFT on human answers fixes that; Alpaca turned 175 seed examples into 52k generated ones for Llama 7B (frame_0008).
• [70:03] RLHF and DPO — SFT clones behavior, is capped by what humans can write, and may cause hallucination; the reward model plus PPO works but is "such a mess" in practice; DPO maximizes preferred outputs directly (frame_0009). RLHF makes answers longer.
• [87:41] Evaluating chat models — perplexity stops working after PPO; Chatbot Arena is the most trusted; AlpacaEval (GPT-4 as judge) has 98% correlation with it for under $10, once corrected for length (GPT-4 told to be verbose wins 64.4% against itself).
• [96:57] Systems — model FLOP utilization (Llama ~45%), 16-bit compute with 32-bit weight updates, operator fusion (frame_0011).

Hook microscope (0-10s)

• Frames: 20 at 2 fps

  Read these hook frames with the Read tool:

  • hook_frames/frame_0001.jpg (t=0.0s)
  • hook_frames/frame_0002.jpg (t=0.5s)
  • hook_frames/frame_0003.jpg (t=1.0s)
  • hook_frames/frame_0004.jpg (t=1.5s)
  • hook_frames/frame_0005.jpg (t=2.0s)
  • hook_frames/frame_0006.jpg (t=2.5s)
  • hook_frames/frame_0007.jpg (t=3.0s)
  • hook_frames/frame_0008.jpg (t=3.5s)
  • hook_frames/frame_0009.jpg (t=4.0s)
  • hook_frames/frame_0010.jpg (t=4.5s)
  • hook_frames/frame_0011.jpg (t=5.0s)
  • hook_frames/frame_0012.jpg (t=5.5s)
  • hook_frames/frame_0013.jpg (t=6.0s)
  • hook_frames/frame_0014.jpg (t=6.5s)
  • hook_frames/frame_0015.jpg (t=7.0s)
  • hook_frames/frame_0016.jpg (t=7.5s)
  • hook_frames/frame_0017.jpg (t=8.0s)
  • hook_frames/frame_0018.jpg (t=8.5s)
  • hook_frames/frame_0019.jpg (t=9.0s)
  • hook_frames/frame_0020.jpg (t=9.5s)

• Word-level transcript (12 words):

  [  5.02s] So
  [  5.86s] let's
  [  6.10s] get
  [  6.26s] started
  [  7.06s] So
  [  7.42s] I'll
  [  7.42s] be
  [  7.80s] talking
  [  7.80s] about
  [  8.18s] building
  [  8.60s] LLMs
  [  9.10s] today

The first 10 seconds are the Stanford Engineering title card (frame_0001), then the room; the only words are "so let's get started" at 0:07. It's a recorded class with no hook in the YouTube sense. What holds attention is the plan stated at 0:56 (five components, and which one he'll skip) and his claim that academia over-invests in architecture.

Editorial profile

• Shots: 12
• Cuts/min: 0.11
• Mean shot length: 522.55s
• Median shot length: 518.13s
• Talking-head ratio: n/a (opencv not installed)

Classroom lecture capture: a fixed camera on the speaker at a podium, cut with full-screen slides (frames 0002, 0005, 0007, 0009, 0011), and student questions throughout. The pacing numbers only reflect 12 transcript-mode samples.

Quotable moments

• [02:38] "In reality, honestly, what matters in practice is mostly the three other topics, so data, evaluation and systems."
• [41:16] "Overfitting doesn't happen with large language models... but for the exam, overfitting exists."
• [55:02] "Don't spend time overcomplicating. Do the simple things, do them well, scale them."
• [66:25] On SFT: "All you do is you tell the model to kind of optimize for one type of user that it saw already in a pretrained data set."
• [85:02] "If you've ever been annoyed at ChatGPT answering you super long sentences, this is because of RLHF."

Entities mentioned

• People: yann-dubois, Richard Sutton, John Schulman
• Companies: Stanford, OpenAI, Anthropic, Google, Meta, Hugging Face, Common Crawl
• Tools / products: ChatGPT, GPT-2, GPT-3, GPT-4, Claude, Gemini, Llama (65B, 2, 3 405B), Alpaca, AlpacaFarm, AlpacaEval, HELM, MMLU, Hugging Face Open LLM Leaderboard, Chatbot Arena, The Pile, C4, Dolma, FineWeb, Open Assistant, LIMA, PPO, DPO, LoRA, PyTorch (torch.compile), H100
• Places: Stanford

Concepts surfaced

• scaling-laws: loss falls predictably with compute, data and parameters on log-log axes; use small runs to choose the architecture and size of the big one. Chinchilla gives 20 tokens per parameter; inference cost pushes production to ~150.
• llm-training-pipeline: pre-training (cross-entropy, tokenizer, filtered web data), SFT, RLHF via PPO or DPO, and evaluation at each stage.
• SFT selects a persona: a pre-trained model already contains every kind of internet writer; a few thousand SFT examples pick one, so more SFT data barely helps (LIMA). The same reading as Karpathy's labeler simulation.
• Pre-training as initialization: post-training is ordinary training started from the pre-trained weights, with a higher learning rate, which is why a tiny dataset can shift the model.
• The bitter lesson (Sutton, 2019): with more compute guaranteed, what matters is architectures that use it, plus data and systems; small architecture tweaks mostly shift the curve's intercept.
• Length bias: humans and LLM judges both prefer longer answers, so RLHF and LLM-judged benchmarks reward verbosity unless corrected.

Transcript

Source: captions.

[00:08] so let's get started uh so I'll be talking about building llms today um so
[00:10] talking about building llms today um so I think a lot of you have heard of llms
[00:13] I think a lot of you have heard of llms before uh but just as a quick recap uh
[00:16] before uh but just as a quick recap uh llms standing for large language models
[00:19] llms standing for large language models are basically all the chat Bots uh that
[00:21] are basically all the chat Bots uh that you've been hearing about recently so uh
[00:24] you've been hearing about recently so uh Chad GPT from open ey Claud from
[00:27] Chad GPT from open ey Claud from entropic Gemini and and lman other type
[00:30] entropic Gemini and and lman other type of models like this and today we'll be
[00:32] of models like this and today we'll be talking about how do they actually work
[00:35] talking about how do they actually work so it's going to be an overview because
[00:36] so it's going to be an overview because it's only one lecture and it's hard to
[00:37] it's only one lecture and it's hard to compress everything but hopefully I'll
[00:39] compress everything but hopefully I'll touch a little bit about all the
[00:41] touch a little bit about all the components that are needed to train uh
[00:42] components that are needed to train uh some of these llms uh also if you have
[00:45] some of these llms uh also if you have questions please interrupt me and ask uh
[00:48] questions please interrupt me and ask uh if you have a question most likely other
[00:50] if you have a question most likely other people in the room or on Zoom have other
[00:53] people in the room or on Zoom have other have the same question so please ask um
[00:57] have the same question so please ask um great so what matters when training llms
[01:00] great so what matters when training llms um so there a few key components that
[01:02] um so there a few key components that matter uh one is the architecture so as
[01:04] matter uh one is the architecture so as you probably all know LMS are newal
[01:07] you probably all know LMS are newal networks and when you think about new
[01:09] networks and when you think about new networks you have to think about what
[01:10] networks you have to think about what architecture you're using and another
[01:12] architecture you're using and another component which is really important uh
[01:14] component which is really important uh is the training loss and the training
[01:16] is the training loss and the training algorithm um so how you actually train
[01:19] algorithm um so how you actually train these models then it's data so uh what
[01:22] these models then it's data so uh what do you train these models on um the
[01:25] do you train these models on um the evaluation which is how do you know
[01:26] evaluation which is how do you know whether you're actually making progress
[01:29] whether you're actually making progress towards the goal of of uh llms and then
[01:32] towards the goal of of uh llms and then the system component so that is like how
[01:34] the system component so that is like how do you actually make these models run on
[01:37] do you actually make these models run on uh Modern Hardware which is really
[01:39] uh Modern Hardware which is really important because these models are
[01:40] important because these models are really large um so now more than ever
[01:43] really large um so now more than ever system is actually really an important
[01:44] system is actually really an important topic um for
[01:46] topic um for llms so those five components um You
[01:50] llms so those five components um You probably all know that llms and if you
[01:52] probably all know that llms and if you don't know LMS are all based on
[01:54] don't know LMS are all based on Transformers or at least some version of
[01:56] Transformers or at least some version of Transformers uh I'm actually not going
[01:58] Transformers uh I'm actually not going to talk about the AR lecture today uh
[02:01] to talk about the AR lecture today uh one because I gave a SE lecture on um
[02:04] one because I gave a SE lecture on um Transformers a few weeks ago and two
[02:07] Transformers a few weeks ago and two because you can find so much information
[02:08] because you can find so much information online on uh Transformers but I think
[02:11] online on uh Transformers but I think you can it's there's much less
[02:13] you can it's there's much less information about the other four topics
[02:14] information about the other four topics so I really want to talk about those um
[02:18] so I really want to talk about those um another thing to say is that most of
[02:19] another thing to say is that most of Academia actually focuses on
[02:21] Academia actually focuses on architecture and training algorithm and
[02:24] architecture and training algorithm and losses um as academics and I've done
[02:26] losses um as academics and I've done that for a lot big part of my career is
[02:29] that for a lot big part of my career is simply we like thinking that this is uh
[02:32] simply we like thinking that this is uh like we make new architectures new
[02:34] like we make new architectures new models and it it seems like it's very
[02:36] models and it it seems like it's very important but in reality honestly what
[02:38] important but in reality honestly what matters in practice is mostly the three
[02:41] matters in practice is mostly the three other topics so data evaluation and
[02:43] other topics so data evaluation and systems uh which is what of most of
[02:46] systems uh which is what of most of Industry actually focuses on um so
[02:49] Industry actually focuses on um so that's also one of the reason why I
[02:50] that's also one of the reason why I don't want to talk too much about the
[02:51] don't want to talk too much about the architecture uh because really the rest
[02:53] architecture uh because really the rest is super
[02:54] is super important um great so overview of the
[02:56] important um great so overview of the lecture I'll be talking about
[02:58] lecture I'll be talking about pre-training so pre-training uh you
[03:00] pre-training so pre-training uh you probably heard that word this is the
[03:01] probably heard that word this is the general word this is kind of the
[03:03] general word this is kind of the classical language modeling uh Paradigm
[03:06] classical language modeling uh Paradigm uh where you basically train your
[03:07] uh where you basically train your language model to essentially model all
[03:09] language model to essentially model all of internet and then there's a post
[03:12] of internet and then there's a post training which is a more recent Paradigm
[03:13] training which is a more recent Paradigm which is taking these large language
[03:15] which is taking these large language models and making them essentially AI
[03:17] models and making them essentially AI assistants um so this is more of a
[03:20] assistants um so this is more of a recent Trend since Chad GPT uh so if you
[03:23] recent Trend since Chad GPT uh so if you ever heard of gpt3 or gpt2 that's really
[03:26] ever heard of gpt3 or gpt2 that's really pre-training land uh if you heard of
[03:28] pre-training land uh if you heard of chat GPT which you probably have this is
[03:30] chat GPT which you probably have this is really posttraining land uh so I'll be
[03:32] really posttraining land uh so I'll be talking about both but I'll start with
[03:34] talking about both but I'll start with pre-training and uh specifically I'll
[03:37] pre-training and uh specifically I'll talk about what is the task of
[03:39] talk about what is the task of pre-training llms and what is the laws
[03:41] pre-training llms and what is the laws that people actually
[03:42] that people actually use so language modeling this is a quick
[03:46] use so language modeling this is a quick recap uh language models at a high level
[03:49] recap uh language models at a high level are simply models of probability
[03:51] are simply models of probability distribution over sequences of tokens or
[03:53] distribution over sequences of tokens or of words so it's basically some uh model
[03:56] of words so it's basically some uh model of P of X1 to XL where X1 is basically
[03:59] of P of X1 to XL where X1 is basically word one and Excel is the last one in
[04:02] word one and Excel is the last one in the sequence or in the sentence um so
[04:04] the sequence or in the sentence um so very concretely if you have a sentence
[04:06] very concretely if you have a sentence like the mouse ate the cheese what the
[04:09] like the mouse ate the cheese what the language model gives you is simply a
[04:11] language model gives you is simply a probability of this sentence being
[04:13] probability of this sentence being uttered by a human or being found on on
[04:16] uttered by a human or being found on on online uh so if you have another
[04:18] online uh so if you have another sentence like the the mouse at cheese uh
[04:22] sentence like the the mouse at cheese uh here there's grammatical mistakes so the
[04:24] here there's grammatical mistakes so the model should know that this uh should
[04:26] model should know that this uh should have some syntactic knowledge so it
[04:28] have some syntactic knowledge so it should know that this has less
[04:29] should know that this has less likelihood of appearing
[04:31] likelihood of appearing online uh if you have another sentence
[04:34] online uh if you have another sentence like the cheese ate the mouse uh then
[04:37] like the cheese ate the mouse uh then the model should hopefully know about
[04:39] the model should hopefully know about the fact that usually cheese don't eat
[04:41] the fact that usually cheese don't eat Mouse um so there's some semantic
[04:43] Mouse um so there's some semantic knowledge and this is less likely than
[04:45] knowledge and this is less likely than the first sentence so this is basically
[04:46] the first sentence so this is basically at a high level what language models are
[04:49] at a high level what language models are um one word that you probably have been
[04:52] um one word that you probably have been hearing a lot in the news are generative
[04:53] hearing a lot in the news are generative models uh so this is just something that
[04:55] models uh so this is just something that can generate models that can generate
[04:57] can generate models that can generate sentences or can generate some data uh
[05:00] sentences or can generate some data uh the reason why we say language models
[05:01] the reason why we say language models are generative models is that once you
[05:03] are generative models is that once you have a model of a distribution you can
[05:05] have a model of a distribution you can simply sample from this model and now we
[05:06] simply sample from this model and now we can generate data uh so you can generate
[05:09] can generate data uh so you can generate sentences uh using a language
[05:12] sentences uh using a language model so the type of models that uh
[05:14] model so the type of models that uh people are all currently using are what
[05:16] people are all currently using are what we call Auto regressive language models
[05:19] we call Auto regressive language models and the key idea of autor regressive
[05:21] and the key idea of autor regressive language models is that you take this
[05:23] language models is that you take this distribution over words and you
[05:26] distribution over words and you basically decompose it into the into the
[05:29] basically decompose it into the into the distribution of the first word multiply
[05:31] distribution of the first word multiply the by the distribution of or the
[05:33] the by the distribution of or the likelihood of the distribution of the
[05:35] likelihood of the distribution of the second word given the first word uh
[05:37] second word given the first word uh multiply by P of the third word given
[05:39] multiply by P of the third word given the first two words um so there's no
[05:42] the first two words um so there's no approximation here this is just the
[05:43] approximation here this is just the chain rule of probability which you
[05:45] chain rule of probability which you hopefully all know about uh really no
[05:47] hopefully all know about uh really no approximation this is just one way of
[05:48] approximation this is just one way of modeling a
[05:50] modeling a distribution uh so slightly more
[05:52] distribution uh so slightly more concisely you can write it as a product
[05:53] concisely you can write it as a product of U of PS of the next word given
[05:57] of U of PS of the next word given everything which happened in the past so
[05:58] everything which happened in the past so of the context and uh so this this is
[06:01] of the context and uh so this this is what we call Auto regressive language
[06:02] what we call Auto regressive language models again this is really not the only
[06:05] models again this is really not the only way of modeling distribution this is
[06:07] way of modeling distribution this is just one way uh it has some benefits and
[06:09] just one way uh it has some benefits and some downsides one downside of
[06:11] some downsides one downside of autoaggressive language models is that
[06:13] autoaggressive language models is that when you actually sample from this
[06:15] when you actually sample from this autoaggressive language model you
[06:16] autoaggressive language model you basically have a for Loop which
[06:18] basically have a for Loop which generates the next word then conditions
[06:20] generates the next word then conditions on that next word and then regenerate an
[06:22] on that next word and then regenerate an other word so basically if you have a
[06:24] other word so basically if you have a longer sentence that you want to
[06:25] longer sentence that you want to generate you it takes more time to
[06:27] generate you it takes more time to generate it uh so there are some
[06:29] generate it uh so there are some downsides of this current Paradigm but
[06:31] downsides of this current Paradigm but that's what we currently have so I'm
[06:33] that's what we currently have so I'm going to talk about this
[06:35] going to talk about this one uh great so Auto regressive language
[06:38] one uh great so Auto regressive language models at a high level um what the task
[06:40] models at a high level um what the task of autoregressive language model is is
[06:42] of autoregressive language model is is simply predicting the next word as I
[06:44] simply predicting the next word as I just said so if you have a sentence like
[06:45] just said so if you have a sentence like she likely prefers uh one potential next
[06:48] she likely prefers uh one potential next word might be dogs and the the way we do
[06:52] word might be dogs and the the way we do it is that we first tokenize so you take
[06:55] it is that we first tokenize so you take these words or subwords you tokenize
[06:57] these words or subwords you tokenize them um and then you give an IDE for
[07:00] them um and then you give an IDE for each token so here you have 1 2 three uh
[07:03] each token so here you have 1 2 three uh then you pass it through this black box
[07:05] then you pass it through this black box as I already said we're not going to
[07:06] as I already said we're not going to talk about the architecture you just
[07:08] talk about the architecture you just pass it pass it through a model and you
[07:10] pass it pass it through a model and you then get a distribution a probability
[07:13] then get a distribution a probability distribution over the next word over the
[07:15] distribution over the next word over the next token and then you sample uh from
[07:19] next token and then you sample uh from this distribution you get a new token
[07:21] this distribution you get a new token and then you DET tokenize so you get a
[07:23] and then you DET tokenize so you get a new ID you then DET toonize and that's
[07:25] new ID you then DET toonize and that's how you basically sample from a language
[07:27] how you basically sample from a language model uh one thing which is important to
[07:29] model uh one thing which is important to not is that the last two TS uh two steps
[07:32] not is that the last two TS uh two steps are actually only need needed during
[07:33] are actually only need needed during inference uh when you do training you
[07:36] inference uh when you do training you just need to predict uh the most likely
[07:38] just need to predict uh the most likely token and you can just compare to the
[07:39] token and you can just compare to the real token which happen in practice and
[07:42] real token which happen in practice and then you basically change the weights of
[07:44] then you basically change the weights of your model to increase the probability
[07:46] your model to increase the probability of generating that
[07:48] of generating that token um great so autoaggressive neural
[07:51] token um great so autoaggressive neural language models so to be slightly more
[07:53] language models so to be slightly more specific still without talking about the
[07:55] specific still without talking about the architecture uh the first thing we do is
[07:57] architecture uh the first thing we do is that we have all of these oh sorry yes
[08:00] that we have all of these oh sorry yes on the previous slide when you're
[08:02] on the previous slide when you're predicting the probability of the next
[08:03] predicting the probability of the next tokens does this mean that your final
[08:05] tokens does this mean that your final like output VOR has to be the same
[08:07] like output VOR has to be the same dimensionality as the number of tokens
[08:09] dimensionality as the number of tokens that you have yes how do you deal with
[08:11] that you have yes how do you deal with like if you have more to like if you're
[08:13] like if you have more to like if you're adding more tokens to your cor something
[08:16] adding more tokens to your cor something yeah so we're going to talk about
[08:18] yeah so we're going to talk about tokenization actually later uh so you
[08:20] tokenization actually later uh so you will get some sense of this you
[08:22] will get some sense of this you basically can deal with adding new
[08:24] basically can deal with adding new tokens I am I'm kind of exaggerating
[08:26] tokens I am I'm kind of exaggerating there are methods for doing it but
[08:28] there are methods for doing it but essentially people don't do it um so
[08:31] essentially people don't do it um so it's really important to think about how
[08:33] it's really important to think about how you tokenize your text and that's why
[08:34] you tokenize your text and that's why we'll talk about that later but it's a
[08:36] we'll talk about that later but it's a very good point to notice that you
[08:37] very good point to notice that you basically the vocabulary size so the
[08:39] basically the vocabulary size so the number of tokens that you have is
[08:40] number of tokens that you have is essentially the output of your uh
[08:42] essentially the output of your uh language model so it's actually pretty
[08:44] language model so it's actually pretty pretty
[08:45] pretty large okay so autoaggressive new
[08:47] large okay so autoaggressive new language models first thing you do is
[08:50] language models first thing you do is that you take every word or every token
[08:52] that you take every word or every token you embed them so you get a um some
[08:55] you embed them so you get a um some Vector representation for each of these
[08:57] Vector representation for each of these tokens um you pass them through some ual
[09:00] tokens um you pass them through some ual Network as we said it's a Transformer
[09:01] Network as we said it's a Transformer then you get a representation for all
[09:04] then you get a representation for all the word in all the words in the context
[09:07] the word in all the words in the context so it's basically representation of the
[09:08] so it's basically representation of the entire sentence uh you pass it through a
[09:11] entire sentence uh you pass it through a linear layer as you just said to
[09:13] linear layer as you just said to basically map it to the number so that
[09:16] basically map it to the number so that the output the number of outputs is the
[09:18] the output the number of outputs is the number of tokens uh you then pass it
[09:20] number of tokens uh you then pass it through some soft Max and you basically
[09:22] through some soft Max and you basically get uh probity distribution over the
[09:26] get uh probity distribution over the next words given every word in the
[09:28] next words given every word in the context
[09:31] context and the law that you use is basically
[09:33] and the law that you use is basically it's essentially a task of classifying
[09:35] it's essentially a task of classifying the next token so it's a very simple
[09:36] the next token so it's a very simple kind of machine learning task so you use
[09:38] kind of machine learning task so you use the cross entry P loss where you
[09:40] the cross entry P loss where you basically you look at the actual Target
[09:44] basically you look at the actual Target that happened which is a target
[09:45] that happened which is a target distribution which is a one hot encoding
[09:47] distribution which is a one hot encoding which here in this in this case says I
[09:49] which here in this in this case says I saw uh the real word that happened is
[09:51] saw uh the real word that happened is cat so that's a one hot um distribution
[09:55] cat so that's a one hot um distribution over cat and here this is the actual uh
[09:58] over cat and here this is the actual uh do you see my mouse oh yeah this is the
[09:59] do you see my mouse oh yeah this is the distribtion that you generated and
[10:01] distribtion that you generated and basically you do cross entropy which
[10:02] basically you do cross entropy which really just increases the probability of
[10:04] really just increases the probability of generating cat and decreases all the the
[10:06] generating cat and decreases all the the probility of generating all the other
[10:07] probility of generating all the other tokens one thing to notice is that as
[10:10] tokens one thing to notice is that as you all know again uh this is just
[10:13] you all know again uh this is just equivalent to maximizing the text log
[10:15] equivalent to maximizing the text log like the text log likelihood because you
[10:17] like the text log likelihood because you can just rewrite the the max over the
[10:20] can just rewrite the the max over the probability of um this autoregressive
[10:23] probability of um this autoregressive language moding task as just being this
[10:25] language moding task as just being this minimum over I just added the log here
[10:28] minimum over I just added the log here and minus which is just the minimum of
[10:30] and minus which is just the minimum of the loss which is the cross enty loss so
[10:32] the loss which is the cross enty loss so basically minimizing the loss is the
[10:33] basically minimizing the loss is the same thing as maximizing the likelihood
[10:35] same thing as maximizing the likelihood of your text any question
[10:44] questions okay
[10:46] okay tokenizer um so this is one thing that
[10:49] tokenizer um so this is one thing that people usually don't talk that much
[10:50] people usually don't talk that much about tokenizers are extremely important
[10:53] about tokenizers are extremely important uh so it's really important that you
[10:55] uh so it's really important that you kind of understand at least uh what they
[10:57] kind of understand at least uh what they do at a high level so why do we need
[10:59] do at a high level so why do we need token in the first place uh first it's
[11:02] token in the first place uh first it's more General than words so one simple
[11:04] more General than words so one simple thing that you might think is oh we're
[11:05] thing that you might think is oh we're just going to take every word that we
[11:07] just going to take every word that we will have you just say every word is a
[11:09] will have you just say every word is a new is a token in its own um but then
[11:12] new is a token in its own um but then what happens is if there's a typo in
[11:14] what happens is if there's a typo in your word then you might not have any
[11:16] your word then you might not have any token associated with this this word
[11:19] token associated with this this word with a typo and then you don't know how
[11:21] with a typo and then you don't know how to actually pass this word with a typo
[11:23] to actually pass this word with a typo into a large language model so what do
[11:25] into a large language model so what do you do next and also even if you think
[11:27] you do next and also even if you think about words words is a very like words
[11:30] about words words is a very like words are fine with like Latin based languages
[11:32] are fine with like Latin based languages uh but if you think about a language
[11:34] uh but if you think about a language like taii you won't have a simple way of
[11:36] like taii you won't have a simple way of tokenizing by spaces because there are
[11:38] tokenizing by spaces because there are no spaces between words um so really uh
[11:41] no spaces between words um so really uh tokens are much more General Than Words
[11:44] tokens are much more General Than Words first thing second thing that you might
[11:45] first thing second thing that you might think is that you might tokenize every
[11:48] think is that you might tokenize every sentence character by character you
[11:50] sentence character by character you might say a is one token b is another
[11:52] might say a is one token b is another token uh that would actually work and
[11:54] token uh that would actually work and probably very well the issue is that
[11:56] probably very well the issue is that then your sequence becomes super long
[11:58] then your sequence becomes super long and as you probably remember from the
[12:00] and as you probably remember from the lecture on on Transformers uh the
[12:03] lecture on on Transformers uh the complexity uh grows quadratically with
[12:06] complexity uh grows quadratically with the length of sequences so you really
[12:07] the length of sequences so you really don't want to have a super long sequence
[12:10] don't want to have a super long sequence um so tokenizers basically try to deal
[12:13] um so tokenizers basically try to deal with those two problems and give common
[12:17] with those two problems and give common subsequences a certain token and usually
[12:20] subsequences a certain token and usually how you should be think about is around
[12:22] how you should be think about is around uh an average every token is around
[12:24] uh an average every token is around three four letters
[12:27] three four letters um and there are many algorithm for
[12:30] um and there are many algorithm for tokenization I'll just talk about one of
[12:31] tokenization I'll just talk about one of them to give you a high level which is
[12:33] them to give you a high level which is what we call bite P en coding which is
[12:34] what we call bite P en coding which is actually pretty common one of the two
[12:36] actually pretty common one of the two most common tokenizers and the way that
[12:38] most common tokenizers and the way that you train a tokenizer is that first you
[12:41] you train a tokenizer is that first you start with a very large Corpus of text
[12:43] start with a very large Corpus of text and here I'm really not talking about
[12:44] and here I'm really not talking about training a large language model yet this
[12:46] training a large language model yet this is purely for the tokenization step uh
[12:48] is purely for the tokenization step uh so this is my large Corpus of text with
[12:50] so this is my large Corpus of text with these five words um then you associate
[12:54] these five words um then you associate every character in this Corpus of text a
[12:57] every character in this Corpus of text a different token uh so here I just split
[13:00] different token uh so here I just split up every character with a different
[13:01] up every character with a different token uh and I just color coded all of
[13:04] token uh and I just color coded all of those tokens and then what you do is
[13:07] those tokens and then what you do is that you go through your text and every
[13:09] that you go through your text and every time you see pairs of tokens that are
[13:12] time you see pairs of tokens that are very common the most common pair of
[13:14] very common the most common pair of token you just merge them so here you
[13:16] token you just merge them so here you see three times the the the tokens T and
[13:19] see three times the the the tokens T and O next to each other so you're just
[13:21] O next to each other so you're just going to say this is a new token and
[13:23] going to say this is a new token and then you continue you repeat that so now
[13:25] then you continue you repeat that so now you have to talk which happens three
[13:28] you have to talk which happens three times to with an E that happens sorry
[13:31] times to with an E that happens sorry two times and an token which happens
[13:34] two times and an token which happens twice and then ex which also happen
[13:37] twice and then ex which also happen twice so this is that if you were to
[13:39] twice so this is that if you were to train a tokenizer on this Corpus of text
[13:42] train a tokenizer on this Corpus of text which is very small that's how you would
[13:44] which is very small that's how you would uh finish with a token with a pre like a
[13:46] uh finish with a token with a pre like a trained tokenizer uh in reality you do
[13:49] trained tokenizer uh in reality you do it on on much larger corpuses of text um
[13:52] it on on much larger corpuses of text um and this is the real tokenizer of uh
[13:55] and this is the real tokenizer of uh actually I think this is gpt3 or chat
[13:57] actually I think this is gpt3 or chat GPT uh and here you see how it would
[13:59] GPT uh and here you see how it would actually separate these words so
[14:01] actually separate these words so basically you see the same thing as what
[14:02] basically you see the same thing as what we gave in the previous example token
[14:05] we gave in the previous example token becomes its own token so tokenizer is
[14:08] becomes its own token so tokenizer is actually split up into two tokens token
[14:11] actually split up into two tokens token and iser um so yeah that's all about
[14:14] and iser um so yeah that's all about tokenizers any questions on that yeah
[14:17] tokenizers any questions on that yeah how do you deal with spes and how do you
[14:18] how do you deal with spes and how do you deal
[14:19] deal with yeah so actually there's a a step
[14:23] with yeah so actually there's a a step before tokenizers which is what we call
[14:24] before tokenizers which is what we call pre- tokenizers which is exactly what
[14:27] pre- tokenizers which is exactly what you just said uh so this is mostly
[14:30] you just said uh so this is mostly in theory there's no reason to deal with
[14:32] in theory there's no reason to deal with spaces and punctuation separately you
[14:34] spaces and punctuation separately you could just say every space gets its own
[14:36] could just say every space gets its own token every um uh punctuation get its
[14:40] token every um uh punctuation get its own token and you can just do all the
[14:42] own token and you can just do all the merging the problem is that so there's
[14:44] merging the problem is that so there's an efficiency question actually training
[14:46] an efficiency question actually training these tokenizes takes a long time uh so
[14:48] these tokenizes takes a long time uh so you better off because you have to
[14:50] you better off because you have to consider every pair of token so what you
[14:52] consider every pair of token so what you end up doing is saying if there's a
[14:54] end up doing is saying if there's a space this is very like pre- tokenizes
[14:56] space this is very like pre- tokenizes are very English specific you say if
[14:58] are very English specific you say if there's a space we're not going to start
[14:59] there's a space we're not going to start looking at the the token that came
[15:01] looking at the the token that came before and the token that came
[15:03] before and the token that came afterwards so you're not merging in
[15:04] afterwards so you're not merging in between spaces but this is just like a
[15:07] between spaces but this is just like a optimiz like a computation optimization
[15:10] optimiz like a computation optimization you could theoretically just deal with
[15:12] you could theoretically just deal with it um the same way as you deal with any
[15:14] it um the same way as you deal with any other character and yeah when you merge
[15:17] other character and yeah when you merge tokens do you delete the tokens that you
[15:19] tokens do you delete the tokens that you merged away or do you keep the the
[15:21] merged away or do you keep the the smaller tokens that merge um you
[15:23] smaller tokens that merge um you actually keep the smaller tokens I mean
[15:25] actually keep the smaller tokens I mean in reality it doesn't matter much
[15:27] in reality it doesn't matter much because um usually on large Corpus of
[15:31] because um usually on large Corpus of text you will have actually everything
[15:33] text you will have actually everything uh but you usually keep the small ones
[15:34] uh but you usually keep the small ones and the reason why you want to do that
[15:36] and the reason why you want to do that is because if in case there's as we said
[15:38] is because if in case there's as we said before you have some um some grammatical
[15:41] before you have some um some grammatical mistakes so some typos you still want to
[15:42] mistakes so some typos you still want to be able to represent these words by
[15:45] be able to represent these words by character um so yeah yes are the tokens
[15:50] character um so yeah yes are the tokens unique so I mean say in this case T Ken
[15:54] unique so I mean say in this case T Ken is there only one occurrence or could do
[15:56] is there only one occurrence or could do you need to leave multiple occurr so
[15:59] you need to leave multiple occurr so they could have take on different
[16:01] they could have take on different meanings or something oh oh I see what
[16:03] meanings or something oh oh I see what you say no no it's every token has its
[16:05] you say no no it's every token has its own uh unique ID um so a usual this is a
[16:10] own uh unique ID um so a usual this is a great question for example if you think
[16:12] great question for example if you think about a bank which could be bank for
[16:14] about a bank which could be bank for like money or bank like water um it will
[16:17] like money or bank like water um it will have the same token but the model will
[16:19] have the same token but the model will learn the Transformer will learn that
[16:21] learn the Transformer will learn that based on the words that are around it it
[16:23] based on the words that are around it it should associate that I'm saying I'm
[16:25] should associate that I'm saying I'm being very high wavy here but associate
[16:27] being very high wavy here but associate that with the with a with a
[16:29] that with the with a with a representation that is either more like
[16:32] representation that is either more like the bank money side or the Bank water
[16:34] the bank money side or the Bank water side um but that's a Transformer that
[16:36] side um but that's a Transformer that does that it's not a
[16:37] does that it's not a tokenizer yes yeah so you mentioned
[16:40] tokenizer yes yeah so you mentioned during tokenization keep the smaller
[16:42] during tokenization keep the smaller tokens you started with right like if
[16:44] tokens you started with right like if you start with a t you keep the T and
[16:46] you start with a t you keep the T and then you build your tokenizer to the
[16:48] then you build your tokenizer to the that you can now in token so let's say
[16:51] that you can now in token so let's say maybe you didn't train on token but like
[16:52] maybe you didn't train on token but like in your data you are trying to encode
[16:54] in your data you are trying to encode token so how does the tokenizer know to
[16:57] token so how does the tokenizer know to encode it with token or
[17:00] encode it with token or a great question you basically when you
[17:02] a great question you basically when you so when you tokenize so that's after
[17:04] so when you tokenize so that's after training of the tokenizer when you
[17:05] training of the tokenizer when you actually apply the tokenizer you
[17:07] actually apply the tokenizer you basically always choose the largest uh
[17:10] basically always choose the largest uh token that you can apply uh so if you
[17:12] token that you can apply uh so if you can do token you will never do T you
[17:14] can do token you will never do T you will always do token um but there's
[17:17] will always do token um but there's actually so people don't usually talk
[17:18] actually so people don't usually talk that much about tokenizers but uh
[17:20] that much about tokenizers but uh there's a lot of of computational
[17:22] there's a lot of of computational benefits uh or computational tricks that
[17:25] benefits uh or computational tricks that you can do for making these things
[17:26] you can do for making these things faster uh so I really don't think we and
[17:29] faster uh so I really don't think we and honestly I think a lot of people think
[17:30] honestly I think a lot of people think that we should just get away from
[17:32] that we should just get away from tokenizers um and just kind of tokenize
[17:34] tokenizers um and just kind of tokenize character by character or bites by bites
[17:37] character by character or bites by bites uh but as I said right now there's this
[17:38] uh but as I said right now there's this issue of like length uh but maybe one
[17:41] issue of like length uh but maybe one day like in five or 10 years we will
[17:43] day like in five or 10 years we will have different architectures that don't
[17:44] have different architectures that don't scale quadratically with the length of
[17:45] scale quadratically with the length of the sequence and uh maybe we'll um yeah
[17:49] the sequence and uh maybe we'll um yeah move away from tokenizes so can you
[17:52] move away from tokenizes so can you share with us the drawback why do people
[17:54] share with us the drawback why do people want to move away from the tokenizer oh
[17:57] want to move away from the tokenizer oh um yeah so think
[18:00] um yeah so think one good example is uh math if you think
[18:04] one good example is uh math if you think about math actually numbers right now
[18:06] about math actually numbers right now are not tokenized so for example 327
[18:09] are not tokenized so for example 327 might have its own token which means
[18:11] might have its own token which means that models when they see numbers they
[18:13] that models when they see numbers they don't see them the same way as we do and
[18:16] don't see them the same way as we do and this is very annoying because what I
[18:17] this is very annoying because what I mean the reason why we can kind of
[18:19] mean the reason why we can kind of generalize with math is because we can
[18:21] generalize with math is because we can deal with every every letter separately
[18:23] deal with every every letter separately and we can then do composition where you
[18:25] and we can then do composition where you know that basically if you add stuff
[18:26] know that basically if you add stuff it's just the same thing as adding every
[18:28] it's just the same thing as adding every one separately plus like whatever the
[18:30] one separately plus like whatever the unit that you add so they can do that um
[18:33] unit that you add so they can do that um so then you have to do like special
[18:34] so then you have to do like special tokenization and like one of the big
[18:37] tokenization and like one of the big changes that GPT 4 did uh is changing
[18:41] changes that GPT 4 did uh is changing the way that they tokenize uh code so
[18:43] the way that they tokenize uh code so for example uh if you have code you know
[18:45] for example uh if you have code you know you have like often in Python these four
[18:47] you have like often in Python these four spaces at the beginning those were dealt
[18:49] spaces at the beginning those were dealt with uh kind of strangely before um and
[18:52] with uh kind of strangely before um and as a result like the model couldn't
[18:54] as a result like the model couldn't really understand uh how to deal with
[18:57] really understand uh how to deal with code uh so so toiz actually a lot um
[19:01] code uh so so toiz actually a lot um okay so I'll move on right now but we
[19:03] okay so I'll move on right now but we can come back later on token Isis great
[19:06] can come back later on token Isis great so we talked about the task the L the
[19:08] so we talked about the task the L the tokenizer let's talk a little bit about
[19:10] tokenizer let's talk a little bit about evaluation uh so the way that LMS are
[19:13] evaluation uh so the way that LMS are usually evaluated is what we call is
[19:15] usually evaluated is what we call is using what we call perplexity um at a
[19:17] using what we call perplexity um at a high level it's basically just your
[19:19] high level it's basically just your validation loss uh the slight difference
[19:21] validation loss uh the slight difference with perplexity is that we use something
[19:23] with perplexity is that we use something that is slightly more interpretable
[19:25] that is slightly more interpretable which is that we use the average per
[19:27] which is that we use the average per token loss and then you expon entiate it
[19:29] token loss and then you expon entiate it and the reason why you exponentiate it
[19:31] and the reason why you exponentiate it is because you want I mean the loss has
[19:34] is because you want I mean the loss has a log inside and you like one humans are
[19:36] a log inside and you like one humans are actually pretty bad at thinking in log
[19:38] actually pretty bad at thinking in log space but two logs depend on the base of
[19:40] space but two logs depend on the base of the log uh while when you exponentiate
[19:43] the log uh while when you exponentiate you basically have everything in the uh
[19:45] you basically have everything in the uh kind of the vocabulary size uh unit um
[19:49] kind of the vocabulary size uh unit um and the average proten is just so that
[19:50] and the average proten is just so that your your complexity is independent of
[19:52] your your complexity is independent of the length of your sequence um so
[19:54] the length of your sequence um so perplexity is just two to the power uh
[19:57] perplexity is just two to the power uh average of the loss of the sequence
[20:00] average of the loss of the sequence um so perplexity is between one and the
[20:03] um so perplexity is between one and the length of the vocabulary of your
[20:05] length of the vocabulary of your tokenizer uh one it's simply well if you
[20:07] tokenizer uh one it's simply well if you predict perfectly the thing which uh
[20:10] predict perfectly the thing which uh every word then every word will have
[20:12] every word then every word will have basically product of ones uh so the best
[20:15] basically product of ones uh so the best perplexity you can have is one if you
[20:17] perplexity you can have is one if you really have no idea you basically
[20:19] really have no idea you basically predict with one divided by uh size of
[20:21] predict with one divided by uh size of vocabulary um and then you do simple
[20:23] vocabulary um and then you do simple math and you basically get perplexity of
[20:25] math and you basically get perplexity of size of vocabulary uh so the intuition
[20:28] size of vocabulary uh so the intuition of perplexity is that basically the
[20:29] of perplexity is that basically the number of tokens that your model is kind
[20:31] number of tokens that your model is kind of hesitating between uh so if you if
[20:34] of hesitating between uh so if you if your model is perfect it doesn't
[20:35] your model is perfect it doesn't hesitate it know exactly the word if it
[20:37] hesitate it know exactly the word if it really has no idea then it hesitates
[20:39] really has no idea then it hesitates between uh all of the
[20:43] between uh all of the vocabulary uh so perplexity really
[20:46] vocabulary uh so perplexity really improved that's perplexity on a standard
[20:48] improved that's perplexity on a standard data set between 2017 and 2023 it it
[20:51] data set between 2017 and 2023 it it went from kind of 70 tokens to less than
[20:54] went from kind of 70 tokens to less than 10 tokens over these five six years so
[20:57] 10 tokens over these five six years so that means that the models were
[20:58] that means that the models were previously as dating between 70 words
[21:00] previously as dating between 70 words every time it was generating a word and
[21:03] every time it was generating a word and now it's as dating between like less
[21:04] now it's as dating between like less than 10 words so that's much better
[21:07] than 10 words so that's much better perplexity is actually not used anymore
[21:09] perplexity is actually not used anymore in academic benchmarking mostly because
[21:11] in academic benchmarking mostly because it depends on the tokenizers that you
[21:12] it depends on the tokenizers that you use uh it depends on the actual data
[21:15] use uh it depends on the actual data that people are evaluating on but it's
[21:16] that people are evaluating on but it's still very important for development of
[21:18] still very important for development of llms so when you when you actually train
[21:20] llms so when you when you actually train your own llm people will still really
[21:22] your own llm people will still really look at the
[21:25] look at the perplexity uh one common other way and
[21:28] perplexity uh one common other way and now more common in Academia of
[21:30] now more common in Academia of evaluating these llms is just by taking
[21:33] evaluating these llms is just by taking all the classical NLP benchmarks and
[21:35] all the classical NLP benchmarks and I'll give you a few examples later and
[21:37] I'll give you a few examples later and just kind of aggregating everything um
[21:39] just kind of aggregating everything um so collect as many automatically
[21:42] so collect as many automatically evaluatable benchmarks and just evaluate
[21:44] evaluatable benchmarks and just evaluate across all of them um so one such if uh
[21:49] across all of them um so one such if uh or actually two such uh benchmarks of
[21:52] or actually two such uh benchmarks of what we call uh Helm which is from
[21:53] what we call uh Helm which is from Stanford and another one is the hugging
[21:55] Stanford and another one is the hugging face open LM leader board which are the
[21:57] face open LM leader board which are the probably two two most common ones right
[21:59] probably two two most common ones right now um so just to give you an idea in
[22:02] now um so just to give you an idea in Helm there are all of these type of
[22:04] Helm there are all of these type of tasks which are mostly things that can
[22:07] tasks which are mostly things that can be easily evaluated uh like question
[22:09] be easily evaluated uh like question answering so think about many different
[22:11] answering so think about many different question answering uh tasks um and the
[22:14] question answering uh tasks um and the benefit with question answering is that
[22:16] benefit with question answering is that you usually know what is the real answer
[22:18] you usually know what is the real answer um so you can the way that you evaluate
[22:20] um so you can the way that you evaluate these models and I'll give you a
[22:21] these models and I'll give you a concrete example in one second um is
[22:23] concrete example in one second um is that you can just look at How likely the
[22:26] that you can just look at How likely the language model is to generate the real
[22:28] language model is to generate the real answer compared to some other answers
[22:31] answer compared to some other answers and that's essentially at a high level
[22:32] and that's essentially at a high level how you evaluate these models um so to
[22:34] how you evaluate these models um so to give you a specific example mlu is
[22:37] give you a specific example mlu is probably the most common um academic
[22:40] probably the most common um academic Benchmark for
[22:41] Benchmark for llms uh and this is just a collection of
[22:45] llms uh and this is just a collection of many question and answers in all of
[22:47] many question and answers in all of those domains for example College
[22:49] those domains for example College medicine College physics astronomy and
[22:51] medicine College physics astronomy and these type of topics and the questions
[22:53] these type of topics and the questions are things like so this in astronomy
[22:55] are things like so this in astronomy what is true for type 1 a supernova then
[22:59] what is true for type 1 a supernova then you give uh four different potential
[23:01] you give uh four different potential answers and you just ask the model which
[23:04] answers and you just ask the model which one is more likely so there are many
[23:06] one is more likely so there are many different ways of doing it either you
[23:07] different ways of doing it either you can look at the likelihood of generating
[23:09] can look at the likelihood of generating all these answers uh or you can ask the
[23:11] all these answers uh or you can ask the model which one is the most likely uh so
[23:13] model which one is the most likely uh so there are different ways that you can
[23:14] there are different ways that you can promp the model but at a high level you
[23:16] promp the model but at a high level you know which one is correct and there are
[23:18] know which one is correct and there are three other mistakes um yes kind
[23:22] three other mistakes um yes kind creating is like unconstrained text as
[23:24] creating is like unconstrained text as the output yeah how do you evaluate a
[23:27] the output yeah how do you evaluate a model if it give something that's you
[23:29] model if it give something that's you know semantically completely identical
[23:33] know semantically completely identical but is not the exact token list that
[23:35] but is not the exact token list that expect yeah so that's a great question
[23:37] expect yeah so that's a great question I'll talk more about that later here in
[23:39] I'll talk more about that later here in this case we don't do unconstrained so
[23:42] this case we don't do unconstrained so the way you would evaluate MML is
[23:44] the way you would evaluate MML is basically either you you ask the first
[23:46] basically either you you ask the first question and then you look at the
[23:48] question and then you look at the likelihood of the model generating a the
[23:51] likelihood of the model generating a the likelihood of the model generating b c
[23:53] likelihood of the model generating b c and d and you look at which one is the
[23:55] and d and you look at which one is the most likely or you can as the model out
[23:57] most likely or you can as the model out of ABC d which one is the most likely
[24:00] of ABC d which one is the most likely and you look at whe the to the most
[24:02] and you look at whe the to the most likely next token is A B C or D so uh
[24:04] likely next token is A B C or D so uh you can strain the model to say it can
[24:06] you can strain the model to say it can only answer these four things you say
[24:09] only answer these four things you say you constraint the model you mean you
[24:12] you constraint the model you mean you constraint The Prompt or do you mean of
[24:14] constraint The Prompt or do you mean of its whole probability distribution
[24:15] its whole probability distribution outputs you only comparing the outputs
[24:19] outputs you only comparing the outputs like you're only comparing the
[24:20] like you're only comparing the a so uh in the second case I gave you
[24:23] a so uh in the second case I gave you you would do exactly the I actually you
[24:25] you would do exactly the I actually you would do both you would prompt the model
[24:26] would do both you would prompt the model saying ABC or D plus you would constrain
[24:29] saying ABC or D plus you would constrain to only uh look at these two these four
[24:31] to only uh look at these two these four tokens in the first case you don't even
[24:33] tokens in the first case you don't even need to generate anything so in the
[24:35] need to generate anything so in the first case you literally just look given
[24:37] first case you literally just look given that it's a language model it can give a
[24:39] that it's a language model it can give a distribution over sentences you just
[24:40] distribution over sentences you just look at what is the likelihood of
[24:43] look at what is the likelihood of generating all of these words what is
[24:45] generating all of these words what is the likelihood of generating the second
[24:47] the likelihood of generating the second choice and you just look at whether the
[24:50] choice and you just look at whether the most likely sentence is actually the
[24:53] most likely sentence is actually the real answer so you don't actually sample
[24:56] real answer so you don't actually sample from it you really just use P of x one
[24:59] from it you really just use P of x one to excel does that make sense uh that
[25:02] to excel does that make sense uh that being said evaluation of open-ended
[25:04] being said evaluation of open-ended questions is something we're going to
[25:06] questions is something we're going to talk about later and is actually really
[25:08] talk about later and is actually really important and really challenging yes
[25:11] important and really challenging yes earlier you mentioned that um like um
[25:13] earlier you mentioned that um like um metrics like flexity are not are not
[25:16] metrics like flexity are not are not like usually used because it depends on
[25:18] like usually used because it depends on like how you do your terization some
[25:20] like how you do your terization some design choices I was wondering if you
[25:22] design choices I was wondering if you could speak more to that oh um yeah so
[25:26] could speak more to that oh um yeah so think about perplexity I told you
[25:27] think about perplexity I told you perplexity is between one and vocabulary
[25:29] perplexity is between one and vocabulary size so now imagine that Chad GPT uses a
[25:33] size so now imagine that Chad GPT uses a tokenizer that has like 10,000 tokens
[25:35] tokenizer that has like 10,000 tokens but Gemini from Google uses a tokenizer
[25:38] but Gemini from Google uses a tokenizer that had 100,000 uh potential tokens
[25:42] that had 100,000 uh potential tokens then actually the Gemini one will will
[25:45] then actually the Gemini one will will have like the upper bound of the the
[25:46] have like the upper bound of the the perplexity that you can get is actually
[25:48] perplexity that you can get is actually worse for Gemini than for Chad GPT does
[25:51] worse for Gemini than for Chad GPT does that make sense so that's just an idea
[25:54] that make sense so that's just an idea it's actually a little bit more
[25:55] it's actually a little bit more complicated than that but that's just
[25:56] complicated than that but that's just like one uh first or the bit of you can
[25:59] like one uh first or the bit of you can see that the tokenizer actually
[26:01] see that the tokenizer actually matters um
[26:05] matters um great okay so evaluation challenges
[26:08] great okay so evaluation challenges there are many I'll just talk about two
[26:10] there are many I'll just talk about two really briefly uh one as I told you
[26:12] really briefly uh one as I told you there are two ways of doing evaluation
[26:14] there are two ways of doing evaluation for these mlu actually there are many
[26:16] for these mlu actually there are many more than two but I give you two
[26:17] more than two but I give you two examples um and it happens that for a
[26:20] examples um and it happens that for a long time even though that was a very
[26:21] long time even though that was a very classical Benchmark that everyone used
[26:24] classical Benchmark that everyone used uh actually different uh different
[26:27] uh actually different uh different companies and different um different uh
[26:30] companies and different um different uh uh different organization were actually
[26:32] uh different organization were actually using different ways of evaluating mlu
[26:35] using different ways of evaluating mlu and as a result you could you get
[26:37] and as a result you could you get completely different results for example
[26:38] completely different results for example Lama
[26:40] Lama 65b uh which was the first model of meta
[26:43] 65b uh which was the first model of meta in the Lama series uh had on Helm 63.7
[26:47] in the Lama series uh had on Helm 63.7 accuracy but on this other um Benchmark
[26:51] accuracy but on this other um Benchmark had like
[26:52] had like 48.8 um so really the way that you
[26:55] 48.8 um so really the way that you evaluate and this is not even talking
[26:56] evaluate and this is not even talking about prompting this is really just kind
[26:58] about prompting this is really just kind of the the way that you evaluate the uh
[27:01] of the the way that you evaluate the uh the models prompting is another issue so
[27:03] the models prompting is another issue so really there are a lot of
[27:04] really there are a lot of inconsistencies it's not as easy as it
[27:06] inconsistencies it's not as easy as it looks uh first thing yeah sorry how can
[27:09] looks uh first thing yeah sorry how can we make sure that all these models AR
[27:11] we make sure that all these models AR trained on The Benchmark okay second
[27:14] trained on The Benchmark okay second thing this is a great question uh chain
[27:16] thing this is a great question uh chain test contamination uh this is something
[27:19] test contamination uh this is something which I would say is really important in
[27:22] which I would say is really important in Academia in uh given that the talk is
[27:25] Academia in uh given that the talk is mostly about training large language
[27:27] mostly about training large language models uh for companies it's maybe not
[27:29] models uh for companies it's maybe not that important CU they know what they
[27:31] that important CU they know what they trained on uh for us we have no idea so
[27:35] trained on uh for us we have no idea so for us it's a real problem uh so there
[27:38] for us it's a real problem uh so there are many different ways of trying to
[27:40] are many different ways of trying to test whether uh the test set sorry
[27:43] test whether uh the test set sorry whether the test set was actually in the
[27:45] whether the test set was actually in the training Set uh one kind of cute trick
[27:49] training Set uh one kind of cute trick um that people uh in in the lab on T lab
[27:53] um that people uh in in the lab on T lab have found is that what you can do is
[27:54] have found is that what you can do is that given that most of the data set
[27:56] that given that most of the data set online are not randomized
[27:59] online are not randomized you can just look at and in that
[28:01] you can just look at and in that language models what they do is just
[28:02] language models what they do is just predict the next word um you can just
[28:04] predict the next word um you can just look at the entire test Set uh what if
[28:07] look at the entire test Set uh what if you generate all the examples in order
[28:11] you generate all the examples in order versus all the examples in a different
[28:13] versus all the examples in a different order and if it's more likely to
[28:15] order and if it's more likely to generate a thing in order given that
[28:17] generate a thing in order given that there's no real order there then it
[28:20] there's no real order there then it means that probably was in a training
[28:21] means that probably was in a training set does that make sense um so there are
[28:24] set does that make sense um so there are many that's like one of them there are
[28:25] many that's like one of them there are many other ways of doing it train test
[28:27] many other ways of doing it train test contamination again not that important
[28:29] contamination again not that important for development really important for
[28:30] for development really important for academic
[28:32] academic benchmarking great so there are many
[28:34] benchmarking great so there are many other challenges but uh I'll move on for
[28:37] other challenges but uh I'll move on for now great data um so data is another
[28:42] now great data um so data is another really big topic um at a high level
[28:44] really big topic um at a high level people just say oh you basically train
[28:46] people just say oh you basically train large language models on all of Internet
[28:49] large language models on all of Internet what does that even mean um so or people
[28:52] what does that even mean um so or people sometimes say all of clean internet
[28:53] sometimes say all of clean internet which is even less defined um so
[28:56] which is even less defined um so internet is very dirty and really not
[28:59] internet is very dirty and really not representative of what we want in
[29:00] representative of what we want in practice if I download a random website
[29:03] practice if I download a random website right now you would be shocked at what
[29:05] right now you would be shocked at what is in there it's definitely not your
[29:07] is in there it's definitely not your Wikipedia um so I'll go really briefly
[29:12] Wikipedia um so I'll go really briefly on like what people do um I can answer
[29:15] on like what people do um I can answer some questions but I mean data is on its
[29:17] some questions but I mean data is on its own is a huge topic uh basically first
[29:20] own is a huge topic uh basically first what you do is download all of Internet
[29:23] what you do is download all of Internet what that means is that you use uh web
[29:25] what that means is that you use uh web crowlers that will go on every web page
[29:27] crowlers that will go on every web page on Internet or every web page that is um
[29:30] on Internet or every web page that is um on Google uh and that is around 250
[29:34] on Google uh and that is around 250 billion pages right now um and that's
[29:37] billion pages right now um and that's around one petabyte of of data so this
[29:40] around one petabyte of of data so this is actually a common common C is one web
[29:42] is actually a common common C is one web crowler so people will usually write
[29:44] crowler so people will usually write their own web crowlers what they do is
[29:46] their own web crowlers what they do is that they use standard web crowlers and
[29:48] that they use standard web crowlers and we common crawl is one of them uh that
[29:50] we common crawl is one of them uh that basically every month adds all the new
[29:53] basically every month adds all the new websites that were added on uh internet
[29:56] websites that were added on uh internet that are found by by Google and they put
[29:58] that are found by by Google and they put it in a big uh basically a big data set
[30:01] it in a big uh basically a big data set um so that's on common call you have
[30:03] um so that's on common call you have around 250 billion pages right now so 1
[30:05] around 250 billion pages right now so 1 E6 gigabytes of data once you have this
[30:09] E6 gigabytes of data once you have this uh so this is a random web page like
[30:12] uh so this is a random web page like literally random uh from this common
[30:14] literally random uh from this common craw and what you see is that one it
[30:16] craw and what you see is that one it really doesn't look at type of things
[30:18] really doesn't look at type of things that you would usually see but actually
[30:19] that you would usually see but actually so this is an HTML page uh it's hard to
[30:22] so this is an HTML page uh it's hard to see but if you look through you will see
[30:25] see but if you look through you will see some content for example here here uh
[30:29] some content for example here here uh tesing world is your ultimate source for
[30:32] tesing world is your ultimate source for the system X high performance server and
[30:34] the system X high performance server and then you have three dots so you don't
[30:36] then you have three dots so you don't even the sentence is not even finished
[30:38] even the sentence is not even finished that's how a random internet looks like
[30:41] that's how a random internet looks like uh so of course it's not that useful if
[30:43] uh so of course it's not that useful if you just train a like large language
[30:44] you just train a like large language model to generate things like this so
[30:46] model to generate things like this so what are some of the steps that are
[30:47] what are some of the steps that are needed first one you extract the text
[30:51] needed first one you extract the text from the HTML so that's what I just try
[30:52] from the HTML so that's what I just try to do by looking at uh basically the
[30:54] to do by looking at uh basically the correct text uh there are a lot of
[30:56] correct text uh there are a lot of challenges by through this for example
[30:58] challenges by through this for example extracting math is actually very
[31:00] extracting math is actually very complicated but pretty important for
[31:02] complicated but pretty important for training large language models um or for
[31:04] training large language models um or for example boiler plates a lot of your
[31:06] example boiler plates a lot of your forums will have the same type of
[31:08] forums will have the same type of headers the same type of Footers uh you
[31:10] headers the same type of Footers uh you don't want to repeat all of this in your
[31:12] don't want to repeat all of this in your data um then you will filter undesirable
[31:16] data um then you will filter undesirable content uh so not safe for work harmful
[31:19] content uh so not safe for work harmful content pii uh so usually every company
[31:22] content pii uh so usually every company has basically a a black list of websites
[31:26] has basically a a black list of websites that they don't want to train the models
[31:27] that they don't want to train the models on that Black List is very long and you
[31:30] on that Black List is very long and you basically say if it comes from there we
[31:31] basically say if it comes from there we don't train on this there are other ways
[31:33] don't train on this there are other ways of doing these things is that you can
[31:34] of doing these things is that you can train a small model for classifying what
[31:37] train a small model for classifying what is pii removing these things um it's
[31:40] is pii removing these things um it's hard every Point here that I'm going to
[31:42] hard every Point here that I'm going to show you is like a hard amount of work
[31:46] show you is like a hard amount of work uh but I'm going to go go quickly
[31:48] uh but I'm going to go go quickly through it so filter undesirable content
[31:50] through it so filter undesirable content second or fourth is the dup D
[31:53] second or fourth is the dup D duplication as I said um you might have
[31:56] duplication as I said um you might have things like headers and Footers in
[31:58] things like headers and Footers in forums that are always the same you want
[32:00] forums that are always the same you want to remove that another thing that you
[32:02] to remove that another thing that you might have is a lot of URLs that are
[32:04] might have is a lot of URLs that are different but actually show the same
[32:07] different but actually show the same website um and you might also have a lot
[32:10] website um and you might also have a lot of like U um paragraphs that come from
[32:13] of like U um paragraphs that come from like common books that are basically
[32:15] like common books that are basically duplicated a thousand times or 10,000
[32:17] duplicated a thousand times or 10,000 times on internet so you have to
[32:19] times on internet so you have to duplicate also very challenging uh
[32:22] duplicate also very challenging uh because you have to do that at scale
[32:24] because you have to do that at scale once you do duplication you will do some
[32:27] once you do duplication you will do some heuristic filtering you will try to
[32:29] heuristic filtering you will try to remove low quality documents uh the way
[32:32] remove low quality documents uh the way you do that are things like rules-based
[32:33] you do that are things like rules-based um filtering for example if you see that
[32:36] um filtering for example if you see that there are some outlier tokens if the
[32:38] there are some outlier tokens if the distribution of tokens in the website is
[32:40] distribution of tokens in the website is very different than the usual
[32:41] very different than the usual distribution of tokens then it's
[32:42] distribution of tokens then it's probably some outlier if you see that
[32:44] probably some outlier if you see that the length of the words in this website
[32:46] the length of the words in this website is super long there's something strange
[32:48] is super long there's something strange going on on that website if you see that
[32:50] going on on that website if you see that the the website has only three words
[32:53] the the website has only three words maybe is it worth training on it maybe
[32:54] maybe is it worth training on it maybe not if it has like 10 million words
[32:57] not if it has like 10 million words maybe there's something also
[32:59] maybe there's something also wrong going on that page um so a lot of
[33:01] wrong going on that page um so a lot of rules like this yes why we filter out
[33:03] rules like this yes why we filter out undesirable content from our dat set
[33:05] undesirable content from our dat set instead of kind
[33:07] instead of kind of putting it in is like a supervised
[33:09] of putting it in is like a supervised loss right like can we not just say like
[33:12] loss right like can we not just say like you know here's this like hate speech
[33:14] you know here's this like hate speech website let's actively try to Let's
[33:18] website let's actively try to Let's actively penalize the for generating
[33:20] actively penalize the for generating we'll do exactly that but not at this
[33:22] we'll do exactly that but not at this step that's where the posttraining will
[33:24] step that's where the posttraining will come from uh pre-training um the idea is
[33:29] come from uh pre-training um the idea is just to say I want to model kind of how
[33:32] just to say I want to model kind of how humans speak essentially um and I want
[33:35] humans speak essentially um and I want to remove all these like headers photos
[33:37] to remove all these like headers photos and and menus and things like this but
[33:39] and and menus and things like this but it's a very good uh like idea that you
[33:41] it's a very good uh like idea that you just had and that's exactly what we'll
[33:43] just had and that's exactly what we'll do
[33:44] do later Next Step modelbased filtering so
[33:47] later Next Step modelbased filtering so once you filtered a lot of data what you
[33:49] once you filtered a lot of data what you will do uh that's actually a very cute
[33:51] will do uh that's actually a very cute trick uh you will take all of Wikipedia
[33:54] trick uh you will take all of Wikipedia and you will look at all the links that
[33:56] and you will look at all the links that are linked through Wikipedia p
[33:59] are linked through Wikipedia p because probably if something is
[34:00] because probably if something is referenced by Wikipedia it's probably
[34:02] referenced by Wikipedia it's probably some high quality website and you will
[34:04] some high quality website and you will train a classifier to predict whether
[34:06] train a classifier to predict whether something comes from whether a document
[34:09] something comes from whether a document comes from one of these references uh
[34:12] comes from one of these references uh from Wikipedia or whether it's from the
[34:14] from Wikipedia or whether it's from the random web and you will try to basically
[34:17] random web and you will try to basically say I want more of the things that come
[34:20] say I want more of the things that come from Wikipedia references does that make
[34:23] from Wikipedia references does that make sense so yeah so you will train a a
[34:25] sense so yeah so you will train a a machine learning uh model usually also
[34:27] machine learning uh model usually also very simp simple models because you need
[34:29] very simp simple models because you need to do that really at scale I mean just
[34:30] to do that really at scale I mean just think about the 250 billion
[34:33] think about the 250 billion Pages uh next one you will try to
[34:36] Pages uh next one you will try to classify your data into different
[34:39] classify your data into different different um domains you will say okay
[34:42] different um domains you will say okay this is entertainment this is books this
[34:44] this is entertainment this is books this is code this is like these type of
[34:46] is code this is like these type of domains and then you will try to either
[34:49] domains and then you will try to either um up or down weight some of the domains
[34:52] um up or down weight some of the domains uh for example you might say uh you
[34:55] uh for example you might say uh you might see that actually if you train
[34:56] might see that actually if you train more on code then actually your model
[34:58] more on code then actually your model becomes bettered on reasoning so that's
[34:59] becomes bettered on reasoning so that's something that people usually say in a
[35:01] something that people usually say in a very handwavy way if you train your
[35:03] very handwavy way if you train your model more code actually it helps
[35:05] model more code actually it helps reasoning so you want to upweight the
[35:07] reasoning so you want to upweight the coding uh distribution because that
[35:09] coding uh distribution because that helps for General language modeling
[35:11] helps for General language modeling skills uh books is usually also another
[35:13] skills uh books is usually also another one that people usually um upweight
[35:16] one that people usually um upweight entertainment they usually downweight uh
[35:19] entertainment they usually downweight uh so things like this of course you want
[35:20] so things like this of course you want to do it so people used to do it maybe
[35:23] to do it so people used to do it maybe uh kind of theistically now there's
[35:26] uh kind of theistically now there's entire pipelines that we'll talk about
[35:28] entire pipelines that we'll talk about of how to do these things uh slightly
[35:30] of how to do these things uh slightly more um
[35:33] more um automatically and then at the end of
[35:35] automatically and then at the end of training uh usually train um after
[35:38] training uh usually train um after training on all of this data that we saw
[35:41] training on all of this data that we saw usually train on very high quality data
[35:43] usually train on very high quality data at the end of of training your large
[35:45] at the end of of training your large language model where you decrease your
[35:47] language model where you decrease your learning rate uh and that basically
[35:49] learning rate uh and that basically means that you're kind of overfitting
[35:50] means that you're kind of overfitting your model on a very high quality data
[35:53] your model on a very high quality data so usually what you do there is like
[35:54] so usually what you do there is like Wikipedia you basically overfit on
[35:57] Wikipedia you basically overfit on Wikipedia yeah and you overfit on like
[36:01] Wikipedia yeah and you overfit on like human uh data that was collected um the
[36:05] human uh data that was collected um the other things like continual pre-training
[36:06] other things like continual pre-training for getting longer context I'm I'm going
[36:08] for getting longer context I'm I'm going to skip over all of these things uh but
[36:10] to skip over all of these things uh but I just to give you a sense of how hard
[36:12] I just to give you a sense of how hard it is when people just say oh I'm going
[36:14] it is when people just say oh I'm going to train on internet that's a lot of
[36:16] to train on internet that's a lot of work um and really we haven't figured it
[36:19] work um and really we haven't figured it out yet so collecting World data is a
[36:23] out yet so collecting World data is a huge part of practical large language
[36:25] huge part of practical large language model uh some might say it's actually
[36:26] model uh some might say it's actually the key yes
[36:28] the key yes about data so basic question so usually
[36:31] about data so basic question so usually when you start with like the terabyte of
[36:33] when you start with like the terabyte of data after I go through all that steps
[36:35] data after I go through all that steps the typical amount of data you have in
[36:38] the typical amount of data you have in and then like how how large a team does
[36:40] and then like how how large a team does it typically think to go through all the
[36:42] it typically think to go through all the steps you talk about so how is the
[36:45] steps you talk about so how is the question how large is the data after you
[36:46] question how large is the data after you filter yeah after you filter and then to
[36:48] filter yeah after you filter and then to go through all the step how large a team
[36:50] go through all the step how large a team do you need to go through like the the
[36:52] do you need to go through like the the other fation sttion uh how slow is it or
[36:57] other fation sttion uh how slow is it or how like how how many people would you
[36:59] how like how how many people would you need to be able to do this uh okay
[37:03] need to be able to do this uh okay that's a great question I'm going to
[37:04] that's a great question I'm going to somewhat answer about the data uh how
[37:07] somewhat answer about the data uh how large is the data set uh at the end of
[37:09] large is the data set uh at the end of this slide uh for number of people that
[37:12] this slide uh for number of people that work on
[37:14] work on it um that's a good question I'm
[37:16] it um that's a good question I'm actually not quite sure but I would
[37:19] actually not quite sure but I would say yeah I actually don't quite no but I
[37:23] say yeah I actually don't quite no but I would say it's probably even bigger than
[37:24] would say it's probably even bigger than the number of people that work on kind
[37:26] the number of people that work on kind of the two tuning of the pre-training of
[37:29] of the two tuning of the pre-training of the model uh so the data is bigger than
[37:32] the model uh so the data is bigger than kind of the modeling aspect um yeah I I
[37:36] kind of the modeling aspect um yeah I I don't think I have a good sense I would
[37:38] don't think I have a good sense I would say probably in Lama's team which have
[37:40] say probably in Lama's team which have like 70 years people I would say maybe
[37:42] like 70 years people I would say maybe 15 work on data uh I yeah all these
[37:47] 15 work on data uh I yeah all these things you don't need that many people
[37:48] things you don't need that many people you need a lot of computer so because
[37:50] you need a lot of computer so because for data you need a lot of CPUs um so
[37:53] for data you need a lot of CPUs um so yeah and I'll answer the second question
[37:55] yeah and I'll answer the second question at the end of this slide so as I just
[37:58] at the end of this slide so as I just kind of alluded to really we haven't
[38:00] kind of alluded to really we haven't solved data at all for pre-training so
[38:02] solved data at all for pre-training so there's a lot of research that that has
[38:04] there's a lot of research that that has to be done first how do you process
[38:06] to be done first how do you process these things super efficiently uh second
[38:08] these things super efficiently uh second how do you balance kind of like all of
[38:09] how do you balance kind of like all of these different domains uh can you do
[38:11] these different domains uh can you do synthetic data generation that's
[38:13] synthetic data generation that's actually a big one right now uh and
[38:15] actually a big one right now uh and because we don't have uh we'll talk
[38:17] because we don't have uh we'll talk about that later we don't have enough
[38:18] about that later we don't have enough data on the internet um can you use
[38:21] data on the internet um can you use multimodal data instead of just text
[38:23] multimodal data instead of just text data and how does that improve even your
[38:26] data and how does that improve even your text performance um
[38:28] text performance um there's a lot of seccy because really
[38:30] there's a lot of seccy because really this is the key of most of the pre-train
[38:33] this is the key of most of the pre-train pre-trained large language models so for
[38:35] pre-trained large language models so for competitive Dynamics uh usually these
[38:37] competitive Dynamics uh usually these these um these companies don't talk
[38:40] these um these companies don't talk about how they do the data collection
[38:42] about how they do the data collection and also there's a copyright liability
[38:44] and also there's a copyright liability issue they definitely don't want to tell
[38:45] issue they definitely don't want to tell you that they've trained on books even
[38:46] you that they've trained on books even though they did um because if not you
[38:48] though they did um because if not you can uh sue them uh common academic
[38:51] can uh sue them uh common academic benchmarks uh so that will kind of
[38:53] benchmarks uh so that will kind of answer what you asked um it started so
[38:56] answer what you asked um it started so those are the smaller ones it's the
[38:58] those are the smaller ones it's the names are not that important but it
[38:59] names are not that important but it started from around 150 billion tokens
[39:02] started from around 150 billion tokens which around uh 800 GB of data now it's
[39:05] which around uh 800 GB of data now it's around 15 trillion of to 15 trillion
[39:07] around 15 trillion of to 15 trillion tokens which is also uh the size of the
[39:11] tokens which is also uh the size of the models that are right now the best
[39:13] models that are right now the best models are probably trained on that
[39:14] models are probably trained on that amount of data so 15 trillion tokens uh
[39:17] amount of data so 15 trillion tokens uh which is probably I guess two order of
[39:20] which is probably I guess two order of manage bigger than that so 80 uh E3 gab
[39:24] manage bigger than that so 80 uh E3 gab so that would be
[39:26] so that would be around 100 to thousand times uh
[39:29] around 100 to thousand times uh filtering of the common crawl if I'm not
[39:31] filtering of the common crawl if I'm not mistaken um so yeah one very one very uh
[39:35] mistaken um so yeah one very one very uh famous one is the pile so this is
[39:38] famous one is the pile so this is academic Benchmark of the pile and we
[39:40] academic Benchmark of the pile and we can just look at what distribution of
[39:42] can just look at what distribution of data they have it's things like um
[39:44] data they have it's things like um archive PBM Central uh which is all the
[39:48] archive PBM Central uh which is all the the biology stuff uh here it's Wikipedia
[39:52] the biology stuff uh here it's Wikipedia you see stack exchange um some GitHub
[39:56] you see stack exchange um some GitHub and some books and things like this um
[39:59] and some books and things like this um again this is on the smaller side so
[40:00] again this is on the smaller side so this is if we look at here this is on
[40:02] this is if we look at here this is on 280b so in reality it's like 100 times
[40:04] 280b so in reality it's like 100 times bigger so you cannot have that much of
[40:06] bigger so you cannot have that much of GitHub and and of
[40:08] GitHub and and of Wikipedia um in terms of close Source
[40:11] Wikipedia um in terms of close Source models just to give you an idea uh Lama
[40:13] models just to give you an idea uh Lama 2 um it was trained on 20 two trillion
[40:16] 2 um it was trained on 20 two trillion tokens lamb 3 15 trillion tokens which
[40:19] tokens lamb 3 15 trillion tokens which is currently the best model that we know
[40:21] is currently the best model that we know on how much it was trained on which is
[40:23] on how much it was trained on which is the same thing as this the the the best
[40:26] the same thing as this the the the best academic or the biggest academic
[40:27] academic or the biggest academic Benchmark which is 15 trillion tokens
[40:29] Benchmark which is 15 trillion tokens GPD 4 we don't really know but it's
[40:31] GPD 4 we don't really know but it's probably in the same water of magnitude
[40:33] probably in the same water of magnitude or it's probably around that actually
[40:34] or it's probably around that actually it's probably around 13 um from leaks if
[40:37] it's probably around 13 um from leaks if the leaks are true
[40:39] the leaks are true um great so scaling laws um any other
[40:44] um great so scaling laws um any other questions on Data before you go to
[40:45] questions on Data before you go to scaling
[40:50] laws sorry I know I'm giving you a lot of information but uh there's a lot into
[40:52] of information but uh there's a lot into training at large language models great
[40:55] training at large language models great scaling laws so so the idea is that what
[40:59] scaling laws so so the idea is that what people saw um around 2020 or at least
[41:02] people saw um around 2020 or at least from a long time but they've been able
[41:03] from a long time but they've been able to kind of theoretically show it or
[41:06] to kind of theoretically show it or impurely show it since 2020 is that the
[41:08] impurely show it since 2020 is that the more data you train your models on and
[41:10] more data you train your models on and the larger the models the better the
[41:12] the larger the models the better the performance this is actually pretty
[41:13] performance this is actually pretty different than what you've seen in this
[41:15] different than what you've seen in this class in this class we teach you about
[41:17] class in this class we teach you about overfitting overfitting doesn't happen
[41:19] overfitting overfitting doesn't happen with large language models uh larger
[41:21] with large language models uh larger models better performance um it's
[41:24] models better performance um it's something that really took a long time
[41:26] something that really took a long time for the community who took this type of
[41:28] for the community who took this type of class to realize um but for the exam
[41:31] class to realize um but for the exam overfitting
[41:33] overfitting exists so okay the idea of scaling laws
[41:37] exists so okay the idea of scaling laws is that if given that you know that more
[41:39] is that if given that you know that more data and larger models will always give
[41:41] data and larger models will always give you better performance can we predict
[41:44] you better performance can we predict how much better your performance will be
[41:47] how much better your performance will be if you increase the amount of data and
[41:48] if you increase the amount of data and the size of your model and surprisingly
[41:51] the size of your model and surprisingly it works uh so here you see three plots
[41:54] it works uh so here you see three plots from a very famous paper called scaling
[41:56] from a very famous paper called scaling loss from openi um here you see on the
[41:59] loss from openi um here you see on the x-axis compute so how much did you train
[42:02] x-axis compute so how much did you train like how much compute did you did you
[42:03] like how much compute did you did you spend for training and here you see test
[42:05] spend for training and here you see test loss so this is essentially I mean it's
[42:07] loss so this is essentially I mean it's not perplexity but it's your validation
[42:09] not perplexity but it's your validation loss um so it's a log of the perplexity
[42:12] loss um so it's a log of the perplexity and if you put these two on uh log scale
[42:15] and if you put these two on uh log scale uh then you see that uh the the
[42:17] uh then you see that uh the the performance or like the this the sorry
[42:20] performance or like the this the sorry the the scaling law is linear uh that
[42:23] the the scaling law is linear uh that means that if you increase your compute
[42:25] means that if you increase your compute by a certain amount you can you can say
[42:27] by a certain amount you can you can say by how much your test loss will actually
[42:29] by how much your test loss will actually decrease same thing with data and same
[42:32] decrease same thing with data and same thing for parameters if you increase the
[42:34] thing for parameters if you increase the data set size your loss will will
[42:37] data set size your loss will will decrease by an amount that is somewhat
[42:39] decrease by an amount that is somewhat predictable if you increase the number
[42:41] predictable if you increase the number of parameters it will decre the loss
[42:43] of parameters it will decre the loss will decrease by amount which is
[42:44] will decrease by amount which is somewhat predictable this is really
[42:47] somewhat predictable this is really amazing um very surprising I mean it
[42:50] amazing um very surprising I mean it looks in nocuous when you look at these
[42:52] looks in nocuous when you look at these type of plots but that's crazy because
[42:53] type of plots but that's crazy because it means that you can predict uh how
[42:56] it means that you can predict uh how well we're going to perform in 2 3 years
[42:58] well we're going to perform in 2 3 years depending on how much compute we will
[43:00] depending on how much compute we will add assuming that these things will hold
[43:02] add assuming that these things will hold there's nothing theoretical about it um
[43:05] there's nothing theoretical about it um yes two things one what is the loss that
[43:08] yes two things one what is the loss that they're using here is this perplexity or
[43:10] they're using here is this perplexity or so it's it's you know I said perplexity
[43:12] so it's it's you know I said perplexity was like two to the power of the LW so
[43:14] was like two to the power of the LW so this is the the the power of the
[43:16] this is the the the power of the perplexity and then the second thing is
[43:19] perplexity and then the second thing is when you like increase the number of
[43:20] when you like increase the number of parameters or you increase the total
[43:22] parameters or you increase the total data set size going dat times doesn't
[43:25] data set size going dat times doesn't that just inherently increase your
[43:27] that just inherently increase your compute like do all this work to
[43:31] compute like do all this work to just specific no this is a great
[43:33] just specific no this is a great question so the compute here is actually
[43:35] question so the compute here is actually a factor of two things the data and the
[43:38] a factor of two things the data and the parameter what I'm showing here is that
[43:39] parameter what I'm showing here is that you can um well actually we're going to
[43:41] you can um well actually we're going to talk about that in details but basically
[43:43] talk about that in details but basically if you increase the number of parameters
[43:44] if you increase the number of parameters you should increase the number of data
[43:46] you should increase the number of data that you have um so you actually don't
[43:49] that you have um so you actually don't go multiple times through the same data
[43:51] go multiple times through the same data set no one does EPO in a lar at least
[43:55] set no one does EPO in a lar at least not yet uh because we have still kind of
[43:59] not yet uh because we have still kind of enough data um so yeah this is all the
[44:01] enough data um so yeah this is all the same Trend which is increase compute
[44:03] same Trend which is increase compute decrease
[44:04] decrease loss yes have we seen the numbers for
[44:08] loss yes have we seen the numbers for the last two years or is it still
[44:10] the last two years or is it still holding it is still holding I I don't
[44:14] holding it is still holding I I don't have like good numbers to show you uh
[44:17] have like good numbers to show you uh but it is still holding
[44:23] surprisingly yes is there no evidence like empirical evidence that you
[44:25] like empirical evidence that you plateau expected PL
[44:29] plateau expected PL no empirical evidence of plateauing
[44:31] no empirical evidence of plateauing anytime soon um why we don't know um
[44:36] anytime soon um why we don't know um will it happen probably I mean it
[44:39] will it happen probably I mean it doesn't need to because it's actually in
[44:40] doesn't need to because it's actually in log scale so it's not like as if it had
[44:43] log scale so it's not like as if it had to go it had to Plateau like
[44:45] to go it had to Plateau like mathematically it could continue
[44:46] mathematically it could continue decreasing like this I mean most people
[44:49] decreasing like this I mean most people think that it will probably Plateau at
[44:50] think that it will probably Plateau at some point we don't know
[44:52] some point we don't know when um okay so that's I'll talk more
[44:56] when um okay so that's I'll talk more about scaling laws now
[44:58] about scaling laws now so why are scaling laws really cool
[45:00] so why are scaling laws really cool imagine that I give you um you're very
[45:03] imagine that I give you um you're very fortunate I gave you 10,000 gpus for
[45:05] fortunate I gave you 10,000 gpus for this month what model will you train how
[45:08] this month what model will you train how do you even go about answering that
[45:09] do you even go about answering that question and I mean this is a a
[45:12] question and I mean this is a a hypothetical but that's exactly what
[45:13] hypothetical but that's exactly what these companies are faced with uh the
[45:16] these companies are faced with uh the old pipeline um which was basically you
[45:20] old pipeline um which was basically you tune High parameters on the big models
[45:22] tune High parameters on the big models so let's say I have 30 days I will train
[45:24] so let's say I have 30 days I will train 30 models for one day each I will pick
[45:27] 30 models for one day each I will pick the best one uh and that will be the
[45:29] the best one uh and that will be the final model that I will use in
[45:31] final model that I will use in production um that means that the model
[45:33] production um that means that the model that I actually used was only trained
[45:35] that I actually used was only trained for one day the new pipeline is that you
[45:39] for one day the new pipeline is that you first find a scaling recipe so you find
[45:41] first find a scaling recipe so you find something that tells you for example oh
[45:44] something that tells you for example oh like one common thing is that if you
[45:45] like one common thing is that if you increase the size of your model you
[45:46] increase the size of your model you should decrease your learning rate so
[45:47] should decrease your learning rate so you find a scaling recipe such that you
[45:49] you find a scaling recipe such that you know if I increase the the the the size
[45:52] know if I increase the the the the size of my model here's what I should do with
[45:54] of my model here's what I should do with some high parameters then you tune your
[45:56] some high parameters then you tune your high parameter
[45:58] high parameter on smaller models of different sizes
[46:01] on smaller models of different sizes let's say I will say for 3 Days of my 30
[46:03] let's say I will say for 3 Days of my 30 days I will train many different models
[46:05] days I will train many different models and I would do highper parameter tuning
[46:07] and I would do highper parameter tuning on these small models each of different
[46:09] on these small models each of different sizes then I will fit a scaling law and
[46:12] sizes then I will fit a scaling law and try to extrapolate from these smaller
[46:14] try to extrapolate from these smaller models which one will be the best if I
[46:18] models which one will be the best if I if I train it for much longer or sorry
[46:20] if I train it for much longer or sorry if I train it for a larger model and
[46:23] if I train it for a larger model and then I will train the final huge model
[46:25] then I will train the final huge model for 27 days instead of just one day
[46:28] for 27 days instead of just one day um so the new pipeline is not train
[46:31] um so the new pipeline is not train things or do high prity tuning on the
[46:33] things or do high prity tuning on the real scale of the model that you're
[46:34] real scale of the model that you're going to use in practice but do things
[46:36] going to use in practice but do things on smaller ones at different scales try
[46:40] on smaller ones at different scales try to predict how well they will perform
[46:42] to predict how well they will perform once you make them bigger I will give I
[46:44] once you make them bigger I will give I will give you a very concrete example
[46:45] will give you a very concrete example right now uh let's say Transformers
[46:48] right now uh let's say Transformers versus lstms let's say you you have
[46:51] versus lstms let's say you you have these 10,000 gpus you will not sure
[46:53] these 10,000 gpus you will not sure which one you should be using should I
[46:54] which one you should be using should I be using Transformer based model or LCM
[46:56] be using Transformer based model or LCM based model what I will do is I will
[46:58] based model what I will do is I will train Transformers at different skills
[47:00] train Transformers at different skills so here you see different parameters on
[47:02] so here you see different parameters on the x-axis Y axis is my test loss I will
[47:05] the x-axis Y axis is my test loss I will then train different different lstms at
[47:07] then train different different lstms at different scales once I have these
[47:09] different scales once I have these points I will see oh it kind of fits a
[47:12] points I will see oh it kind of fits a scaling law I will fit my scaling law
[47:14] scaling law I will fit my scaling law and then I will be able to predict oh if
[47:17] and then I will be able to predict oh if I had 10 times more compute here's how
[47:19] I had 10 times more compute here's how well I would perform for the LM it's
[47:22] well I would perform for the LM it's actually slightly less linear for the
[47:23] actually slightly less linear for the lstm but like you could probably try to
[47:25] lstm but like you could probably try to predict where you would end up and
[47:27] predict where you would end up and clearly from this plot you would see
[47:29] clearly from this plot you would see that Transformers are better um one
[47:31] that Transformers are better um one thing to notice when you read these type
[47:33] thing to notice when you read these type of scaling laws is that are two things
[47:34] of scaling laws is that are two things that are important uh one is really your
[47:38] that are important uh one is really your scaling rate uh which is kind of the uh
[47:42] scaling rate uh which is kind of the uh the slope of the the slope of the
[47:45] the slope of the the slope of the scaling law the other thing is your um
[47:48] scaling law the other thing is your um your intercept like you could start
[47:51] your intercept like you could start worse but actually become better over
[47:53] worse but actually become better over time it just happens that lstms are
[47:55] time it just happens that lstms are worse for both uh but I could show you
[47:57] worse for both uh but I could show you another one where things you can predict
[48:00] another one where things you can predict that actually after a certain scale
[48:01] that actually after a certain scale you're better off using that type of
[48:03] you're better off using that type of model than others uh so that's why
[48:05] model than others uh so that's why scaling laws are actually really
[48:08] scaling laws are actually really useful any questions on
[48:16] that yeah so these are all kind of very how how sensitive are these to like
[48:17] how how sensitive are these to like small differences in the architecture
[48:19] small differences in the architecture like one one like Transformer
[48:22] like one one like Transformer architecture versus another Transformer
[48:23] architecture versus another Transformer architecture you basically have to like
[48:25] architecture you basically have to like fit your own curve and make basically
[48:27] fit your own curve and make basically say like oh scaling law has tell me
[48:28] say like oh scaling law has tell me there should be some like logarithmic
[48:31] there should be some like logarithmic function let me extrapolate that for my
[48:34] function let me extrapolate that for my own yeah so uh usually for example if
[48:38] own yeah so uh usually for example if you're an academic and you want to now
[48:39] you're an academic and you want to now at least that's like pretty recent and
[48:41] at least that's like pretty recent and you want to propose a new like
[48:43] you want to propose a new like activation uh that's exactly what you
[48:45] activation uh that's exactly what you will do you will fit a scaling law show
[48:47] will do you will fit a scaling law show another scaling law with the standard
[48:48] another scaling law with the standard like I don't know G and you will say
[48:50] like I don't know G and you will say that it's better in reality once you
[48:52] that it's better in reality once you start thinking about it in scaling loss
[48:53] start thinking about it in scaling loss terms you really realize that actually
[48:56] terms you really realize that actually all the architecture differences that we
[48:57] all the architecture differences that we can make like the small minor ones all
[48:59] can make like the small minor ones all they do is maybe change a little bit the
[49:02] they do is maybe change a little bit the The
[49:02] The Intercept but really that doesn't matter
[49:05] Intercept but really that doesn't matter uh cuz just train it for 10 hours longer
[49:07] uh cuz just train it for 10 hours longer or like wait for the next uh for the
[49:09] or like wait for the next uh for the next Compu gpus and these things are
[49:11] next Compu gpus and these things are really secondary which is exactly why I
[49:13] really secondary which is exactly why I was telling you originally people spend
[49:15] was telling you originally people spend too much time on the architecture and
[49:16] too much time on the architecture and losses um in reality these things don't
[49:18] losses um in reality these things don't matter as much data though if you use
[49:20] matter as much data though if you use good data you will have much better
[49:22] good data you will have much better scaling loss than if use bad data so
[49:25] scaling loss than if use bad data so that really matters
[49:27] that really matters uh another really cool thing you can do
[49:29] uh another really cool thing you can do with scaling laws is that you can ask
[49:31] with scaling laws is that you can ask yourself uh how to optimally allocate
[49:34] yourself uh how to optimally allocate training resources should I train larger
[49:36] training resources should I train larger models because we saw that it's better
[49:38] models because we saw that it's better when you train larger models but we saw
[49:40] when you train larger models but we saw that it's also better when you use more
[49:42] that it's also better when you use more data so which one should I do should I
[49:44] data so which one should I do should I just train on more data a smaller model
[49:46] just train on more data a smaller model or should I train a larger model on less
[49:48] or should I train a larger model on less data um so chinchilla is a very famous
[49:52] data um so chinchilla is a very famous paper that first showed this uh the way
[49:54] paper that first showed this uh the way they did it I want to give you a little
[49:56] they did it I want to give you a little bit of a sense of what these plots are
[49:58] bit of a sense of what these plots are uh here you see training loss again on
[50:00] uh here you see training loss again on the x-axis you see parameter parameter
[50:02] the x-axis you see parameter parameter differences uh sorry parameter size uh
[50:04] differences uh sorry parameter size uh number of parameters so the size of the
[50:05] number of parameters so the size of the model and here all these curves are what
[50:08] model and here all these curves are what we call isof flops which is that all the
[50:11] we call isof flops which is that all the models on this curve H have been trained
[50:15] models on this curve H have been trained with the same amount of
[50:16] with the same amount of compute um the way that you do that is
[50:18] compute um the way that you do that is that you train you change sorry you vary
[50:21] that you train you change sorry you vary the number of tokens that we trained on
[50:23] the number of tokens that we trained on and the size of the models but you vary
[50:24] and the size of the models but you vary in such a way that the total compute is
[50:26] in such a way that the total compute is constant
[50:27] constant okay so all these curves that you see
[50:29] okay so all these curves that you see with different colors have different
[50:31] with different colors have different amount of computers that were trained on
[50:33] amount of computers that were trained on then you take the best one for each of
[50:34] then you take the best one for each of those curves once you have the best one
[50:37] those curves once you have the best one for each of those curves um you can ask
[50:41] for each of those curves um you can ask you can plot um how much flops it was
[50:44] you can plot um how much flops it was and which curve were you on and how much
[50:46] and which curve were you on and how much parameters did you actually use for
[50:49] parameters did you actually use for training that specific point you put
[50:51] training that specific point you put that on the on the log log uh scale
[50:55] that on the on the log log uh scale again and now you fit a scaling law
[50:56] again and now you fit a scaling law again so now I have something which
[50:59] again so now I have something which tells me if I want to train a model of
[51:01] tells me if I want to train a model of 10^ 23 flops here's exactly the number
[51:04] 10^ 23 flops here's exactly the number of parameters that I should be using 100
[51:07] of parameters that I should be using 100 100b and you can do the same thing with
[51:09] 100b and you can do the same thing with flops and
[51:11] flops and tokens so now you can predict if if I
[51:14] tokens so now you can predict if if I tell you exactly I have one month of
[51:16] tell you exactly I have one month of compute what size of model should I be
[51:18] compute what size of model should I be training F your scaling law and I tell
[51:21] training F your scaling law and I tell you um of course that all looks
[51:23] you um of course that all looks beautiful in reality like there's like
[51:25] beautiful in reality like there's like there's a lot of like small things of
[51:27] there's a lot of like small things of like should you be counting like
[51:28] like should you be counting like embedding parameters like there's
[51:30] embedding parameters like there's there's a lot of complexities but if you
[51:31] there's a lot of complexities but if you do things well these things actually do
[51:34] do things well these things actually do hold um so the optimal number of
[51:37] hold um so the optimal number of parameters that that chinchilla Pap have
[51:39] parameters that that chinchilla Pap have found is to use 20 tokens for every
[51:42] found is to use 20 tokens for every parameter that you train uh so if you
[51:44] parameter that you train uh so if you add one more parameter you should add
[51:46] add one more parameter you should add you should train your thing on your
[51:47] you should train your thing on your model on 20 more tokens so one caveat
[51:51] model on 20 more tokens so one caveat here is that this is optimal training
[51:52] here is that this is optimal training resources so that is telling me if you
[51:55] resources so that is telling me if you have 10^ 23 FL
[51:57] have 10^ 23 FL or if you have like 100 I don't know how
[51:59] or if you have like 100 I don't know how much that is100 million or 10 no that's
[52:02] much that is100 million or 10 no that's much less actually let's say I have $5
[52:04] much less actually let's say I have $5 million to to train my best model that
[52:06] million to to train my best model that gets the lowest loss how how what would
[52:09] gets the lowest loss how how what would I train on in reality these companies
[52:11] I train on in reality these companies need to think about inference also if
[52:13] need to think about inference also if you have a smaller model they will spend
[52:16] you have a smaller model they will spend less over time um so actually if you
[52:19] less over time um so actually if you consider the inference cost you have
[52:20] consider the inference cost you have other papers that Tred to show that um
[52:23] other papers that Tred to show that um it's around
[52:24] it's around 150 uh parameters per sorry tokens per
[52:28] 150 uh parameters per sorry tokens per parameters because you prefer having a
[52:29] parameters because you prefer having a smaller model cuz over time you're going
[52:32] smaller model cuz over time you're going to you're going to actually um spend
[52:35] to you're going to actually um spend less money on inference of these models
[52:38] less money on inference of these models so 150 to one that's around what the
[52:41] so 150 to one that's around what the best models are trained on right now at
[52:43] best models are trained on right now at least the ones that are that are used um
[52:47] least the ones that are that are used um in practice for in
[52:49] in practice for in production
[52:51] production great any question on
[52:59] chin great oh sorry in practice how expensive is inference for these models
[53:01] expensive is inference for these models rela to
[53:03] rela to train actually very expensive uh I will
[53:06] train actually very expensive uh I will not talk about inference because that
[53:07] not talk about inference because that would be another entire lecture but just
[53:10] would be another entire lecture but just think about Chad GPT where they have I
[53:13] think about Chad GPT where they have I don't know how much it is now like 600
[53:15] don't know how much it is now like 600 million people that used it um like
[53:20] million people that used it um like that's a lot
[53:22] that's a lot um yeah so it's actually very expensive
[53:25] um yeah so it's actually very expensive there's a lot of optimization you can do
[53:26] there's a lot of optimization you can do for in though um and that's an entire
[53:28] for in though um and that's an entire other lecture so I'm going to skip that
[53:30] other lecture so I'm going to skip that uh this time but it's very
[53:33] uh this time but it's very interesting okay tuning um as I said
[53:36] interesting okay tuning um as I said there are many things that you can uh
[53:37] there are many things that you can uh answer with scaling laws I just try to
[53:39] answer with scaling laws I just try to give you two examples uh but really
[53:42] give you two examples uh but really there are many things what data do you
[53:43] there are many things what data do you use what mixture what data mixing
[53:46] use what mixture what data mixing waiting you use data mixtures that's
[53:47] waiting you use data mixtures that's what we talked about before uh what
[53:49] what we talked about before uh what architecture you use whether you should
[53:51] architecture you use whether you should make your models uh wider or deeper um
[53:54] make your models uh wider or deeper um should you be paying for more gpus or
[53:57] should you be paying for more gpus or actually collecting more data um all
[53:59] actually collecting more data um all these things are things you can try to
[54:01] these things are things you can try to answer with scaling
[54:02] answer with scaling laws one thing I want to say is the bit
[54:05] laws one thing I want to say is the bit lesson if you ever heard of Richard
[54:07] lesson if you ever heard of Richard sudden a very famous blog post in 2019
[54:11] sudden a very famous blog post in 2019 um what he realized uh which I think not
[54:16] um what he realized uh which I think not enough people realize I didn't
[54:17] enough people realize I didn't definitely did not realize at that time
[54:19] definitely did not realize at that time um is that once you see these type of
[54:22] um is that once you see these type of scaling laws you know that the more
[54:23] scaling laws you know that the more compute you have the better models you
[54:26] compute you have the better models you will get so with skill you will get
[54:27] will get so with skill you will get better model and you also know by Mo law
[54:30] better model and you also know by Mo law or these type of variant of Mo law that
[54:32] or these type of variant of Mo law that you will always have better compute then
[54:34] you will always have better compute then the only thing that matters is just to
[54:37] the only thing that matters is just to have architectures that can leverage
[54:39] have architectures that can leverage computation so what matters is basically
[54:42] computation so what matters is basically systems data and less so the
[54:44] systems data and less so the architecture like the small architecture
[54:46] architecture like the small architecture differences like your your your
[54:48] differences like your your your activation and things like this uh so I
[54:50] activation and things like this uh so I think that's like one of the reasons why
[54:51] think that's like one of the reasons why most of research focuses on um some
[54:55] most of research focuses on um some things that for industry matters less
[54:57] things that for industry matters less and I was one of those researchers for a
[54:59] and I was one of those researchers for a large part of my my career um so don't
[55:03] large part of my my career um so don't spend time over complicating do the
[55:05] spend time over complicating do the simple things do it well seal them
[55:08] simple things do it well seal them that's really what openi taught us with
[55:11] that's really what openi taught us with um with chat gpg and with all the gpts
[55:14] um with chat gpg and with all the gpts before okay I want to give you some
[55:17] before okay I want to give you some backup the envelope computation so I
[55:19] backup the envelope computation so I might be off by a few factors here but I
[55:21] might be off by a few factors here but I just want to give you a sense of how
[55:23] just want to give you a sense of how costly it is to train some of these
[55:24] costly it is to train some of these models I'll give as an example
[55:27] models I'll give as an example Lama 3 400b which is currently the best
[55:29] Lama 3 400b which is currently the best open source model that you can get uh it
[55:32] open source model that you can get uh it was trained on 15.6 tokens it has 45
[55:37] was trained on 15.6 tokens it has 45 billion parameters so just now that you
[55:39] billion parameters so just now that you know what is like this uh optimal tokens
[55:42] know what is like this uh optimal tokens per parameter that's around 40 so that's
[55:44] per parameter that's around 40 so that's a little bit more than chinchilla but
[55:46] a little bit more than chinchilla but less than this like inference uh optimal
[55:49] less than this like inference uh optimal um model so they went for training
[55:52] um model so they went for training optimality uh flops for this model so
[55:55] optimality uh flops for this model so one simple uh way to compute flops is
[55:58] one simple uh way to compute flops is six uh times the number of parameters
[56:01] six uh times the number of parameters times the number of data you train on uh
[56:03] times the number of data you train on uh so if you do the simple calculation here
[56:05] so if you do the simple calculation here it's 3.8 e25 flops the reason why this
[56:08] it's 3.8 e25 flops the reason why this is important is that if you follow the
[56:10] is important is that if you follow the little bit the news there's an executive
[56:12] little bit the news there's an executive order from Biden that basically says
[56:14] order from Biden that basically says that once you have uh 1 e26 parameters
[56:18] that once you have uh 1 e26 parameters uh sorry flops uh then you have special
[56:20] uh sorry flops uh then you have special scrutiny on your models so they went 2x
[56:23] scrutiny on your models so they went 2x less than that so they really went right
[56:25] less than that so they really went right below this to not have special scrutiny
[56:27] below this to not have special scrutiny so 38 uh I might be off by a little bit
[56:30] so 38 uh I might be off by a little bit but it's definitely under the 1
[56:40] 26 oh um so paramet p is parameters n is data number of tokens this is a uh this
[56:44] data number of tokens this is a uh this is just an
[56:45] is just an approximation we
[56:47] approximation we yeah okay uh compute and we know that
[56:51] yeah okay uh compute and we know that they trained on 16,000
[56:53] they trained on 16,000 h100s um and we know the throughput but
[56:57] h100s um and we know the throughput but they they said it too uh so if you do
[56:59] they they said it too uh so if you do the computation it takes around 70 days
[57:02] the computation it takes around 70 days um or 26 million GPU hours at least
[57:06] um or 26 million GPU hours at least that's with my uh back of the envelope
[57:08] that's with my uh back of the envelope computation they actually said that they
[57:09] computation they actually said that they use 30 million instead of 26 million GPU
[57:13] use 30 million instead of 26 million GPU hours um so maybe they had like some uh
[57:17] hours um so maybe they had like some uh some challenges I don't really know but
[57:19] some challenges I don't really know but if you follow the simple computation
[57:20] if you follow the simple computation it's around 70 days um cost uh I mean
[57:25] it's around 70 days um cost uh I mean this it's hard to to approximate but I'm
[57:27] this it's hard to to approximate but I'm just going to say it's kind of the rent
[57:29] just going to say it's kind of the rent like what if I were to rent h100s that
[57:32] like what if I were to rent h100s that many h100s for that many days how much
[57:35] many h100s for that many days how much will I pay uh h100 a lower bound on the
[57:39] will I pay uh h100 a lower bound on the on the renting uh cost of h100 is around
[57:41] on the renting uh cost of h100 is around 2 hours uh $2 per hour so if you
[57:44] 2 hours uh $2 per hour so if you multiply this by 26 million uh hours uh
[57:48] multiply this by 26 million uh hours uh you get 52 million uh dollars so they
[57:51] you get 52 million uh dollars so they probably pay less than that but not
[57:54] probably pay less than that but not actually much less because all these um
[57:57] actually much less because all these um all these services that actually rent
[57:59] all these services that actually rent gpus they don't make that much money so
[58:01] gpus they don't make that much money so it's it's probably slightly less but not
[58:03] it's it's probably slightly less but not that much less um now salary I said 50
[58:07] that much less um now salary I said 50 employees 500k per
[58:10] employees 500k per year say yeah it's probably the right
[58:12] year say yeah it's probably the right ballpark 25 million uh so if you put all
[58:14] ballpark 25 million uh so if you put all together around 75 million um dollars
[58:18] together around 75 million um dollars for
[58:18] for training uh this Slammer model I'm
[58:21] training uh this Slammer model I'm probably off by like 10 million but but
[58:23] probably off by like 10 million but but that's kind of right uh bpk
[58:28] that's kind of right uh bpk carbon emitted um a lot of people might
[58:30] carbon emitted um a lot of people might ask like also the cost is not the only
[58:33] ask like also the cost is not the only thing that is important so I did the
[58:34] thing that is important so I did the computation um it's around 4 uh 4,000 um
[58:40] computation um it's around 4 uh 4,000 um tons of CO2 equivalent that is actually
[58:44] tons of CO2 equivalent that is actually only 2,000 return tickets from JFK to uh
[58:47] only 2,000 return tickets from JFK to uh London so right now uh carbon emitted is
[58:50] London so right now uh carbon emitted is actually not uh I mean it's huge but
[58:53] actually not uh I mean it's huge but it's not like um meaningful yeah yet I
[58:57] it's not like um meaningful yeah yet I think in maybe GPT 6 gpt7 once you
[59:01] think in maybe GPT 6 gpt7 once you multiply this by 100 that might become a
[59:04] multiply this by 100 that might become a real issue right now it's still not uh I
[59:06] real issue right now it's still not uh I think um an issue in the grand scheme of
[59:09] think um an issue in the grand scheme of things next model the way you should be
[59:11] things next model the way you should be thinking about these models is that
[59:13] thinking about these models is that every new generation the number of flops
[59:15] every new generation the number of flops essentially uh multiplies 10x or at
[59:18] essentially uh multiplies 10x or at least that's what they try uh if they
[59:19] least that's what they try uh if they have enough energy and if they can buy
[59:21] have enough energy and if they can buy enough
[59:22] enough gpus uh great any question on these back
[59:25] gpus uh great any question on these back of the envelope math
[59:31] no okay so now we talked about pre-training
[59:35] okay so now we talked about pre-training I wanted to also chat about systems
[59:37] I wanted to also chat about systems because now we know computer is really
[59:38] because now we know computer is really important so there's a question of how
[59:40] important so there's a question of how do you optimize the how do you optimize
[59:42] do you optimize the how do you optimize your computer I will leave that for the
[59:44] your computer I will leave that for the end because I'm not sure how much time
[59:46] end because I'm not sure how much time we will have I think it's important but
[59:47] we will have I think it's important but hopefully I I'll be able to to talk
[59:49] hopefully I I'll be able to to talk about it later it's slightly different
[59:52] about it later it's slightly different than what we've been talking about right
[59:54] than what we've been talking about right now so I'll move on to post training for
[59:56] now so I'll move on to post training for now
[59:57] now so the task of post training ER the
[59:59] so the task of post training ER the reason why we need to do Post training
[01:00:01] reason why we need to do Post training is as I told you before um it's to make
[01:00:05] is as I told you before um it's to make AI assistants so language modeling is
[01:00:08] AI assistants so language modeling is not uh really the thing that you want
[01:00:10] not uh really the thing that you want when you have an AI assistant uh for
[01:00:13] when you have an AI assistant uh for example if you ask to gbd3 which is a
[01:00:15] example if you ask to gbd3 which is a purely language Model A pure language
[01:00:17] purely language Model A pure language model not a um not an aligned one if you
[01:00:20] model not a um not an aligned one if you ask a question like explain the moon
[01:00:22] ask a question like explain the moon landing to a
[01:00:24] landing to a six-year-old the completion that you
[01:00:26] six-year-old the completion that you would get is something like explain the
[01:00:27] would get is something like explain the theory of gravity to a six-year-old
[01:00:29] theory of gravity to a six-year-old because what it learned is that on on on
[01:00:31] because what it learned is that on on on internet if you have one question you
[01:00:33] internet if you have one question you usually have maybe another bullet point
[01:00:35] usually have maybe another bullet point of other similar questions you don't
[01:00:37] of other similar questions you don't usually have question and then answer
[01:00:38] usually have question and then answer later uh this is not what you want from
[01:00:41] later uh this is not what you want from an AI assistant so how do we uh do this
[01:00:45] an AI assistant so how do we uh do this alignment which is this post training
[01:00:47] alignment which is this post training and making these models
[01:00:49] and making these models assistance um so the goal of this
[01:00:51] assistance um so the goal of this alignment is to basically get LMS follow
[01:00:54] alignment is to basically get LMS follow the instructions that are given um by
[01:00:57] the instructions that are given um by users and and maybe some designers kind
[01:01:01] users and and maybe some designers kind of desires um so think about moderation
[01:01:04] of desires um so think about moderation you don't want the model like open ey
[01:01:06] you don't want the model like open ey definitely doesn't want the model to say
[01:01:07] definitely doesn't want the model to say stuff that is very
[01:01:09] stuff that is very toxic um so here you see on the left
[01:01:11] toxic um so here you see on the left hand side uh that when you ask a
[01:01:13] hand side uh that when you ask a question it actually provides a a real
[01:01:15] question it actually provides a a real answer so it's not like uh before the
[01:01:17] answer so it's not like uh before the llm and on the right hand side you see
[01:01:20] llm and on the right hand side you see that it would if you ask to write a
[01:01:22] that it would if you ask to write a tweet describing how a certain part of
[01:01:25] tweet describing how a certain part of the population are evil it will say that
[01:01:27] the population are evil it will say that it cannot do that um so that's kind of
[01:01:31] it cannot do that um so that's kind of this
[01:01:32] this alignment uh the background here is that
[01:01:36] alignment uh the background here is that uh basically the data that you want for
[01:01:39] uh basically the data that you want for training some of these models um is like
[01:01:42] training some of these models um is like we know what we want which is just
[01:01:43] we know what we want which is just asking humans this is a question this is
[01:01:45] asking humans this is a question this is the answer that you want uh but the
[01:01:47] the answer that you want uh but the thing is that it's very expensive to
[01:01:48] thing is that it's very expensive to collect that data and it's hard to find
[01:01:50] collect that data and it's hard to find it online uh in contrast pre-training
[01:01:53] it online uh in contrast pre-training data is not what you want but there's a
[01:01:55] data is not what you want but there's a lot of it um so what what we will do a
[01:01:58] lot of it um so what what we will do a the main idea is simply take a pre-train
[01:02:00] the main idea is simply take a pre-train large language model pre-train all of
[01:02:02] large language model pre-train all of internet and then you just fine tune so
[01:02:04] internet and then you just fine tune so you just change a little bit of weights
[01:02:05] you just change a little bit of weights on the type of data that you actually
[01:02:07] on the type of data that you actually want and hopefully given it you already
[01:02:09] want and hopefully given it you already pre-train it on all of Internet it
[01:02:10] pre-train it on all of Internet it basically learns or knows how to speak
[01:02:13] basically learns or knows how to speak in English and and knows a standard um
[01:02:17] in English and and knows a standard um language syntax uh then you can really
[01:02:20] language syntax uh then you can really find tune in with very little
[01:02:22] find tune in with very little data okay sft so supervis fine tuning is
[01:02:26] data okay sft so supervis fine tuning is really exactly what I just said which is
[01:02:28] really exactly what I just said which is the idea of fine-tuning the large
[01:02:29] the idea of fine-tuning the large language model on uh basically the
[01:02:32] language model on uh basically the desired answers that are collected from
[01:02:34] desired answers that are collected from humans um so why is it called supervis
[01:02:37] humans um so why is it called supervis fine tuning because you basically want
[01:02:39] fine tuning because you basically want to do language modeling on the real
[01:02:41] to do language modeling on the real ansers so language modeling is this like
[01:02:43] ansers so language modeling is this like next word prediction and and that's the
[01:02:45] next word prediction and and that's the fine-tuning part and then you want to do
[01:02:47] fine-tuning part and then you want to do it on desired answers given by humans so
[01:02:49] it on desired answers given by humans so that's why we call it
[01:02:50] that's why we call it supervis so how do we collect this data
[01:02:53] supervis so how do we collect this data well we I just said it you just ask
[01:02:55] well we I just said it you just ask humans uh to to tell you this is the
[01:02:57] humans uh to to tell you this is the this is a question this is the answer
[01:02:58] this is a question this is the answer that you uh you would want from some of
[01:03:00] that you uh you would want from some of these models so this is an example um
[01:03:03] these models so this is an example um sorry I can't read very well on my
[01:03:05] sorry I can't read very well on my computer but uh my kid uh needs to do a
[01:03:08] computer but uh my kid uh needs to do a science um no let's read this one can
[01:03:10] science um no let's read this one can you write a short introduction about the
[01:03:12] you write a short introduction about the relevance of the term monopsony and then
[01:03:14] relevance of the term monopsony and then it says monopsony refers to a market
[01:03:16] it says monopsony refers to a market structure blah blah blah and that's a
[01:03:17] structure blah blah blah and that's a human that wrote that um so actually
[01:03:20] human that wrote that um so actually this is open Assistant which was a a way
[01:03:22] this is open Assistant which was a a way to collect um uh data online by
[01:03:27] to collect um uh data online by humans so this type of supervised fine
[01:03:31] humans so this type of supervised fine tuning or alignment is really the key of
[01:03:33] tuning or alignment is really the key of Chad GPT this is what made uh the big
[01:03:36] Chad GPT this is what made uh the big jump from gpt3 which was mostly
[01:03:38] jump from gpt3 which was mostly something that was known by AI
[01:03:39] something that was known by AI researchers to Chad GPT which became
[01:03:42] researchers to Chad GPT which became known by basically
[01:03:44] known by basically everyone
[01:03:46] everyone um so the problem with uh human data is
[01:03:52] um so the problem with uh human data is that it's uh very slow to collect and
[01:03:54] that it's uh very slow to collect and very expensive um so
[01:03:57] very expensive um so one possible simple idea is to use llms
[01:04:01] one possible simple idea is to use llms to scale data collection uh so that's
[01:04:04] to scale data collection uh so that's exactly what we did with alpaca uh one
[01:04:06] exactly what we did with alpaca uh one year ago what we did is that we asked uh
[01:04:09] year ago what we did is that we asked uh humans or we use a data set of human uh
[01:04:11] humans or we use a data set of human uh question answers so there were 175 uh
[01:04:14] question answers so there were 175 uh question answers here and we asked the
[01:04:16] question answers here and we asked the best mod at the time so text3 to
[01:04:19] best mod at the time so text3 to basically generate many more of these
[01:04:21] basically generate many more of these question and answers so all we did is
[01:04:23] question and answers so all we did is like this is what humans would write now
[01:04:25] like this is what humans would write now write similar answers and similar
[01:04:27] write similar answers and similar questions and we collected 52,000 LM
[01:04:31] questions and we collected 52,000 LM generated question answers and then what
[01:04:33] generated question answers and then what we did is simply we took Lama 7B which
[01:04:35] we did is simply we took Lama 7B which was the best pre-train model at the time
[01:04:37] was the best pre-train model at the time and we just fine- tuned this with
[01:04:38] and we just fine- tuned this with supervised fine tuning as I told you and
[01:04:40] supervised fine tuning as I told you and that's how we got um the Alpac s7b
[01:04:44] that's how we got um the Alpac s7b model uh and this is the type of data
[01:04:46] model uh and this is the type of data that we collected so things like what
[01:04:48] that we collected so things like what does algorithm mean an algorithm is a
[01:04:50] does algorithm mean an algorithm is a step by a stepbystep uh set of
[01:04:53] step by a stepbystep uh set of instruction used to solve a problem or
[01:04:55] instruction used to solve a problem or achieve a goal blah blah blah blah so
[01:04:57] achieve a goal blah blah blah blah so the data is not actually it's actually
[01:04:58] the data is not actually it's actually pretty good given it was LM generated by
[01:05:00] pretty good given it was LM generated by LMS from essentially two generations ago
[01:05:04] LMS from essentially two generations ago um so that really started at least for
[01:05:07] um so that really started at least for us kind of as an academic replication of
[01:05:09] us kind of as an academic replication of chat GPT uh now it really there's a big
[01:05:13] chat GPT uh now it really there's a big field of like synthetic data generation
[01:05:15] field of like synthetic data generation of how to use llms to basically make
[01:05:18] of how to use llms to basically make development of llms faster um and by
[01:05:22] development of llms faster um and by basically by decreasing the amount of of
[01:05:24] basically by decreasing the amount of of human hours that you need
[01:05:27] human hours that you need quantity of data so we talked about what
[01:05:30] quantity of data so we talked about what type of data and how we collect it um
[01:05:32] type of data and how we collect it um one thing which is surprising with sft
[01:05:34] one thing which is surprising with sft is that you don't need that much data uh
[01:05:37] is that you don't need that much data uh so what this paper showed this is called
[01:05:38] so what this paper showed this is called Lima is that if you have if you scale
[01:05:41] Lima is that if you have if you scale the amount of data that use from uh
[01:05:43] the amount of data that use from uh supervised fine training from 2,000 to
[01:05:46] supervised fine training from 2,000 to 32,000 it really doesn't help much so
[01:05:48] 32,000 it really doesn't help much so here scaling laws definitely don't help
[01:05:50] here scaling laws definitely don't help um so the the intuition here is that all
[01:05:53] um so the the intuition here is that all you learn um is is you learn how to
[01:05:57] you learn um is is you learn how to format your desired answers another way
[01:05:59] format your desired answers another way of saying it is that your pre-trained
[01:06:01] of saying it is that your pre-trained models they essentially model the
[01:06:04] models they essentially model the distribution of every user on internet
[01:06:06] distribution of every user on internet one that might write bullet points
[01:06:08] one that might write bullet points another one that might answer qu answer
[01:06:10] another one that might answer qu answer question with an answer so all you tell
[01:06:12] question with an answer so all you tell your model is like wait you should
[01:06:14] your model is like wait you should actually be optimizing more for this
[01:06:16] actually be optimizing more for this type of user than another one so you're
[01:06:17] type of user than another one so you're not actually teaching it and you're not
[01:06:19] not actually teaching it and you're not teaching anything through this um sft uh
[01:06:23] teaching anything through this um sft uh so supervis fine tuning all you do is
[01:06:25] so supervis fine tuning all you do is you tell the model to kind of optimize
[01:06:27] you tell the model to kind of optimize for one type of user that it saw already
[01:06:29] for one type of user that it saw already in a pre-train data set so the knowledge
[01:06:32] in a pre-train data set so the knowledge is already in the pre-train llm uh and
[01:06:34] is already in the pre-train llm uh and you basically just specialize to one
[01:06:35] you basically just specialize to one type of
[01:06:37] type of user great any question on
[01:06:40] user great any question on sft yes so I know it's a big issue with
[01:06:44] sft yes so I know it's a big issue with synthetic data where uh if you keep
[01:06:48] synthetic data where uh if you keep generating data from the same
[01:06:49] generating data from the same distribution eventually you're not
[01:06:50] distribution eventually you're not learning a new distribution you're
[01:06:52] learning a new distribution you're essentially playing with it it just
[01:06:53] essentially playing with it it just bootstrapping that yeah surely
[01:06:56] bootstrapping that yeah surely you can't scale that forever right you
[01:06:58] you can't scale that forever right you can't keep going on and generating from
[01:07:00] can't keep going on and generating from the same distribution you hope to learn
[01:07:01] the same distribution you hope to learn something new yeah uh so are there it's
[01:07:04] something new yeah uh so are there it's an active area of research but any
[01:07:06] an active area of research but any thoughts that you have around how people
[01:07:08] thoughts that you have around how people are maybe thinking around this and uh
[01:07:10] are maybe thinking around this and uh better ways to bootstrap or to give up
[01:07:12] better ways to bootstrap or to give up on this idea and and realize that the
[01:07:14] on this idea and and realize that the chart shows you don't need that many so
[01:07:16] chart shows you don't need that many so just get humans to generate 2,000 really
[01:07:18] just get humans to generate 2,000 really good uh yeah so that's a very good
[01:07:21] good uh yeah so that's a very good question uh so for the data stuff so I'm
[01:07:23] question uh so for the data stuff so I'm saying it's not that important for sft
[01:07:25] saying it's not that important for sft but there will be another thing we'll
[01:07:26] but there will be another thing we'll talk about right after where actually
[01:07:28] talk about right after where actually data does
[01:07:29] data does matter my intuition based on not that
[01:07:33] matter my intuition based on not that much empirical results is that you can
[01:07:35] much empirical results is that you can still get um even though you use your
[01:07:38] still get um even though you use your LMS if you use purely LM generated text
[01:07:41] LMS if you use purely LM generated text and you do that for like three four
[01:07:42] and you do that for like three four generations of llms I agree with you
[01:07:44] generations of llms I agree with you that probably you won't improve much but
[01:07:46] that probably you won't improve much but for me what is important is how do you
[01:07:47] for me what is important is how do you use like human in the loop with llms not
[01:07:50] use like human in the loop with llms not purely LMS not purely uh humans but
[01:07:53] purely LMS not purely uh humans but maybe what you can do is just have the
[01:07:55] maybe what you can do is just have the model generate some new text and just uh
[01:07:58] model generate some new text and just uh humans write a few Edits edits are much
[01:08:00] humans write a few Edits edits are much faster than writing the entire text and
[01:08:02] faster than writing the entire text and I think that if you have that type of
[01:08:03] I think that if you have that type of collaboration then from like kind of an
[01:08:06] collaboration then from like kind of an information theoretical point of view
[01:08:07] information theoretical point of view you still get additional information but
[01:08:09] you still get additional information but you still much faster than if you use
[01:08:11] you still much faster than if you use humans and I think that as a field we'll
[01:08:13] humans and I think that as a field we'll probably move towards these type of
[01:08:15] probably move towards these type of things uh which is um really just
[01:08:17] things uh which is um really just finding the examples that are important
[01:08:20] finding the examples that are important and and asking humans it's kind of
[01:08:22] and and asking humans it's kind of active learning just asking humans
[01:08:23] active learning just asking humans exactly when uh you need to to get
[01:08:27] exactly when uh you need to to get inputs yes do we train with like the
[01:08:30] inputs yes do we train with like the same loss function the same like General
[01:08:32] same loss function the same like General training algorithm for the supervis
[01:08:34] training algorithm for the supervis tuning bit as we do for the for the
[01:08:36] tuning bit as we do for the for the pre-training right because like the
[01:08:38] pre-training right because like the examples you showed I think the the
[01:08:40] examples you showed I think the the important thing of the good examples is
[01:08:44] important thing of the good examples is they're like supera accurate there's
[01:08:46] they're like supera accurate there's these more complex still just like chain
[01:08:49] these more complex still just like chain same so that's why here I yeah I didn't
[01:08:51] same so that's why here I yeah I didn't maybe didn't emphasize enough this is
[01:08:53] maybe didn't emphasize enough this is just language modeling fine tun the LM
[01:08:55] just language modeling fine tun the LM with language model on the desired
[01:08:56] with language model on the desired answers so this is literally the same
[01:08:58] answers so this is literally the same loss um it will be different in two
[01:09:01] loss um it will be different in two seconds but the first step of sft is
[01:09:04] seconds but the first step of sft is literally the same loss where you just
[01:09:06] literally the same loss where you just say Okay I want to actually specialize
[01:09:07] say Okay I want to actually specialize on that type of data so there's even a
[01:09:09] on that type of data so there's even a question of like what is pre-training
[01:09:11] question of like what is pre-training what is post-training because in reality
[01:09:12] what is post-training because in reality it's just like a different data that you
[01:09:13] it's just like a different data that you use the reason why we usually call it
[01:09:15] use the reason why we usually call it post training is that the way we collect
[01:09:16] post training is that the way we collect that data is very
[01:09:18] that data is very different great great questions uh yes
[01:09:22] different great great questions uh yes maybe it's the same question but why
[01:09:24] maybe it's the same question but why would these 2,000 examples have such an
[01:09:27] would these 2,000 examples have such an overweighted
[01:09:28] overweighted influence you tun so that's why we uh
[01:09:32] influence you tun so that's why we uh also that's another reason why we call
[01:09:33] also that's another reason why we call it post training is that we use
[01:09:34] it post training is that we use different type of hyper parameters so
[01:09:36] different type of hyper parameters so you know I told you basically at the end
[01:09:37] you know I told you basically at the end of pre training you essentially end up
[01:09:39] of pre training you essentially end up with a learning rate of zero and here
[01:09:41] with a learning rate of zero and here you're going to increase your learning
[01:09:42] you're going to increase your learning rate so like 1 eus 5 one E Yeah and and
[01:09:45] rate so like 1 eus 5 one E Yeah and and so um the weight that you give to them
[01:09:48] so um the weight that you give to them is actually
[01:09:49] is actually different
[01:09:52] different um okay uh Second Step or second part of
[01:09:56] um okay uh Second Step or second part of this post training um is what we call
[01:09:59] this post training um is what we call reinforcement learning from Human
[01:10:01] reinforcement learning from Human feedback or rhf uh some of you might
[01:10:03] feedback or rhf uh some of you might have heard of that um the idea is that
[01:10:06] have heard of that um the idea is that sft has a problem namely that uh you do
[01:10:09] sft has a problem namely that uh you do behavioral cloning which means that you
[01:10:11] behavioral cloning which means that you just try to clone what the humans would
[01:10:14] just try to clone what the humans would say and that had that has many issues
[01:10:17] say and that had that has many issues one of them is that you're bound by
[01:10:18] one of them is that you're bound by human abilities so if um like humans
[01:10:24] human abilities so if um like humans actually humans won't generate the
[01:10:26] actually humans won't generate the things that they think is actually the
[01:10:27] things that they think is actually the best thing to generate so if you ask me
[01:10:30] best thing to generate so if you ask me to write a book I mean I can definitely
[01:10:32] to write a book I mean I can definitely enjoy a book I can probably say one book
[01:10:33] enjoy a book I can probably say one book is better than another but I'm
[01:10:35] is better than another but I'm definitely not going to be as good as
[01:10:36] definitely not going to be as good as writing the book that I want to read uh
[01:10:38] writing the book that I want to read uh so you're going to be bound by the human
[01:10:40] so you're going to be bound by the human ability to generate things even though
[01:10:41] ability to generate things even though the humans might be better at
[01:10:42] the humans might be better at distinguishing between things that's one
[01:10:44] distinguishing between things that's one issue issue number two uh I find that
[01:10:47] issue issue number two uh I find that actually pretty interesting is that it
[01:10:49] actually pretty interesting is that it might if you ever heard of the word
[01:10:50] might if you ever heard of the word hallucination so this is llms generating
[01:10:53] hallucination so this is llms generating F like false information
[01:10:56] F like false information hallucination might these people have um
[01:10:59] hallucination might these people have um hypothesized that that can come from the
[01:11:01] hypothesized that that can come from the supervised fine tuning even if you do
[01:11:03] supervised fine tuning even if you do supervised fine tuning on data that is
[01:11:05] supervised fine tuning on data that is correct and the reason why that is is
[01:11:08] correct and the reason why that is is that if uh given I told you that
[01:11:11] that if uh given I told you that basically sftt is with very little data
[01:11:13] basically sftt is with very little data and it's with data that doesn't the
[01:11:16] and it's with data that doesn't the model doesn't learn anything new so what
[01:11:18] model doesn't learn anything new so what if the human gives an answer that the
[01:11:21] if the human gives an answer that the model didn't know was true from the
[01:11:24] model didn't know was true from the model perspective you the human
[01:11:26] model perspective you the human basically is telling the the model uh
[01:11:28] basically is telling the the model uh generate this thing that seems plausible
[01:11:31] generate this thing that seems plausible but actually have no idea if it's true
[01:11:33] but actually have no idea if it's true or not um so just to give you a very
[01:11:36] or not um so just to give you a very concrete example if we go back to this
[01:11:38] concrete example if we go back to this uh monopsony example can you write blah
[01:11:40] uh monopsony example can you write blah blah blah about monopsony uh imagine
[01:11:42] blah blah about monopsony uh imagine that a human uh wrote a reference on
[01:11:45] that a human uh wrote a reference on this type of book um and that book might
[01:11:48] this type of book um and that book might exist that might be a correct reference
[01:11:49] exist that might be a correct reference but what if the llm never saw this
[01:11:51] but what if the llm never saw this reference during pre-training then it
[01:11:53] reference during pre-training then it doesn't know that it's a correct
[01:11:54] doesn't know that it's a correct reference so really what you tell the
[01:11:56] reference so really what you tell the model is to generate or make up some
[01:11:58] model is to generate or make up some plausibly sounding reference um rather
[01:12:01] plausibly sounding reference um rather than actually tell the real reference
[01:12:04] than actually tell the real reference that it saw during pre-training uh so
[01:12:06] that it saw during pre-training uh so hallucination might be um uh a re like
[01:12:11] hallucination might be um uh a re like might be caused by this sft that's
[01:12:13] might be caused by this sft that's problem number two does that all make
[01:12:15] problem number two does that all make sense great problem number three price
[01:12:18] sense great problem number three price generating the ideal answers is very
[01:12:21] generating the ideal answers is very pricey and that comes back to your
[01:12:23] pricey and that comes back to your question um of like humans writing
[01:12:25] question um of like humans writing answer is actually pretty
[01:12:27] answer is actually pretty expensive um so that's where rhf comes
[01:12:30] expensive um so that's where rhf comes in the idea is that instead of cloning
[01:12:32] in the idea is that instead of cloning the behaviors of humans we're going to
[01:12:35] the behaviors of humans we're going to maximize human preference um and the way
[01:12:38] maximize human preference um and the way we're going to do that so the pipeline
[01:12:40] we're going to do that so the pipeline is that for a certain for every
[01:12:42] is that for a certain for every instruction you're going to ask a model
[01:12:43] instruction you're going to ask a model to generate two answers um and usually
[01:12:47] to generate two answers um and usually use a pretty good model so you usually
[01:12:49] use a pretty good model so you usually don't use an LM here you use a sft uh
[01:12:52] don't use an LM here you use a sft uh fine tune you use a fine tuned llm
[01:12:54] fine tune you use a fine tuned llm already to give like pretty good answers
[01:12:57] already to give like pretty good answers and then you ask labelers which of these
[01:13:00] and then you ask labelers which of these two answers was better so select the
[01:13:02] two answers was better so select the preferred one and then with different
[01:13:04] preferred one and then with different type of algorithms we're going to talk
[01:13:06] type of algorithms we're going to talk about the algorithms um you just
[01:13:07] about the algorithms um you just fine-tune the model to generate more of
[01:13:09] fine-tune the model to generate more of the green thing than the red thing so
[01:13:11] the green thing than the red thing so more of the good stuff uh so now the
[01:13:13] more of the good stuff uh so now the question is how and we're going to talk
[01:13:15] question is how and we're going to talk about that right
[01:13:16] about that right now so there are two ways that we're
[01:13:19] now so there are two ways that we're going to talk about and two that are
[01:13:21] going to talk about and two that are mainly used in the community um the
[01:13:24] mainly used in the community um the first one is simply the idea of of using
[01:13:25] first one is simply the idea of of using reinforcement learning so hopefully you
[01:13:27] reinforcement learning so hopefully you all know what reinforcement learning is
[01:13:29] all know what reinforcement learning is now um so when you think about using
[01:13:33] now um so when you think about using reinforcement learning one important
[01:13:34] reinforcement learning one important question is like what is the reward that
[01:13:35] question is like what is the reward that we're optimizing uh so in this case
[01:13:37] we're optimizing uh so in this case there are really two options that I
[01:13:39] there are really two options that I could think about the first one you
[01:13:40] could think about the first one you could just say I'm going to compare the
[01:13:42] could just say I'm going to compare the output generated by some baseline the
[01:13:44] output generated by some baseline the output generated by my model U and I'm
[01:13:47] output generated by my model U and I'm just going to ask the human to say which
[01:13:49] just going to ask the human to say which one is better and I'm going to use this
[01:13:51] one is better and I'm going to use this as a reward so if I'm better than the
[01:13:53] as a reward so if I'm better than the Baseline this is a plus one if not it's
[01:13:55] Baseline this is a plus one if not it's a minus one one uh so now it's binary
[01:13:57] a minus one one uh so now it's binary reward the problem with binary reward is
[01:13:59] reward the problem with binary reward is that it's very sparse and you don't get
[01:14:00] that it's very sparse and you don't get much information out of it uh like maybe
[01:14:03] much information out of it uh like maybe your answer was slightly better maybe it
[01:14:05] your answer was slightly better maybe it was like way better and you don't really
[01:14:07] was like way better and you don't really know from this um how much better it was
[01:14:11] know from this um how much better it was so option two is that you can train what
[01:14:13] so option two is that you can train what we call a reward model which is simply a
[01:14:16] we call a reward model which is simply a classifier uh so you use machine
[01:14:18] classifier uh so you use machine learning to to classify how much better
[01:14:21] learning to to classify how much better uh two outputs are from the preference
[01:14:25] uh two outputs are from the preference from the perspective of the human um so
[01:14:28] from the perspective of the human um so this is a little bit meta but what you
[01:14:29] this is a little bit meta but what you basically do is that you train uh you
[01:14:31] basically do is that you train uh you take um a reward model R which is a uh
[01:14:35] take um a reward model R which is a uh just a large also a large um a large
[01:14:38] just a large also a large um a large classifier and you basically ask this
[01:14:41] classifier and you basically ask this reward model you give it the input and
[01:14:43] reward model you give it the input and the actual output that you have one of
[01:14:45] the actual output that you have one of the two outputs uh and you just um
[01:14:48] the two outputs uh and you just um exponentiate that so that's the soft Max
[01:14:49] exponentiate that so that's the soft Max law that you all know about and now you
[01:14:51] law that you all know about and now you divide by um the the exponential
[01:14:56] divide by um the the exponential reward uh on the first example sorry on
[01:14:59] reward uh on the first example sorry on the first output and this is on the
[01:15:01] the first output and this is on the second output and you basically train so
[01:15:03] second output and you basically train so the reason why you do that is that you
[01:15:04] the reason why you do that is that you train your your model you train this
[01:15:06] train your your model you train this reward model to be able to classify um
[01:15:10] reward model to be able to classify um how much better one output is to another
[01:15:13] how much better one output is to another one so another uh slightly less
[01:15:15] one so another uh slightly less convoluted way of saying it is that your
[01:15:17] convoluted way of saying it is that your reward model will output some reward
[01:15:20] reward model will output some reward that will be used as the logits of your
[01:15:22] that will be used as the logits of your soft Max so now if you have high logic
[01:15:25] soft Max so now if you have high logic in your softmax it means that you highly
[01:15:28] in your softmax it means that you highly likely this um output is
[01:15:32] likely this um output is better uh so that's what we call Bradley
[01:15:34] better uh so that's what we call Bradley ter model yes is this reward model going
[01:15:37] ter model yes is this reward model going over the entire output or is it
[01:15:39] over the entire output or is it going um so this takes the
[01:15:44] going um so this takes the entire uh yeah this takes the entire
[01:15:46] entire uh yeah this takes the entire output at once so it takes all the input
[01:15:48] output at once so it takes all the input and all the output and it gives one
[01:15:50] and all the output and it gives one number
[01:15:56] yes would human be sorry with the reward model where would a human be like oh I
[01:15:59] model where would a human be like oh I see okay sorry maybe I wasn't clear um
[01:16:03] see okay sorry maybe I wasn't clear um you train this reward model to fit this
[01:16:07] you train this reward model to fit this green and and red preference from humans
[01:16:10] green and and red preference from humans so basically you train a classifier to
[01:16:12] so basically you train a classifier to say whether the humans prefer red or
[01:16:15] say whether the humans prefer red or green uh but instead of using the binary
[01:16:18] green uh but instead of using the binary reward which is what the human would
[01:16:19] reward which is what the human would tell you you basically use the logits of
[01:16:22] tell you you basically use the logits of the soft Max and the thing with the
[01:16:24] the soft Max and the thing with the logits is that that logits are
[01:16:26] logits is that that logits are continuous so now you know that if your
[01:16:28] continuous so now you know that if your reward model said it has high logits
[01:16:31] reward model said it has high logits then in some ways the human highly
[01:16:33] then in some ways the human highly prefer this answer to some other
[01:16:36] prefer this answer to some other answer great um so as I just said
[01:16:40] answer great um so as I just said continuous information so it's better so
[01:16:42] continuous information so it's better so that's what people uh use in practice or
[01:16:44] that's what people uh use in practice or at least used to use in practice I'll
[01:16:46] at least used to use in practice I'll tell you about uh the other algorithm
[01:16:48] tell you about uh the other algorithm later uh so what you do at the end is
[01:16:50] later uh so what you do at the end is that you basically try to just use
[01:16:52] that you basically try to just use reinforcement learning that you know
[01:16:53] reinforcement learning that you know about now we know we have reward what
[01:16:56] about now we know we have reward what you sample through is the generation
[01:16:58] you sample through is the generation from your large language model um and
[01:17:00] from your large language model um and then you just use some regularization
[01:17:02] then you just use some regularization term so the reason why you do this
[01:17:03] term so the reason why you do this regularization term is for avoiding what
[01:17:05] regularization term is for avoiding what we call over optimization so this reward
[01:17:07] we call over optimization so this reward model might not be really represent like
[01:17:10] model might not be really represent like might not perfectly model human
[01:17:11] might not perfectly model human preferences so you don't want to
[01:17:13] preferences so you don't want to maximize this thing to essentially
[01:17:15] maximize this thing to essentially Infinity um and you do it using uh po
[01:17:20] Infinity um and you do it using uh po which is a common uh reinforcement
[01:17:23] which is a common uh reinforcement learning algorithm um one thing to note
[01:17:26] learning algorithm um one thing to note here because it will be important for
[01:17:27] here because it will be important for later is that when we use maximum
[01:17:30] later is that when we use maximum likelihood
[01:17:32] likelihood um sorry now the large language models
[01:17:35] um sorry now the large language models are actually a policy for your
[01:17:37] are actually a policy for your reinforcement learning it's not
[01:17:39] reinforcement learning it's not maximizing maximum likelihood anymore
[01:17:41] maximizing maximum likelihood anymore which means that you're not modeling any
[01:17:43] which means that you're not modeling any distribution anymore and the reason why
[01:17:45] distribution anymore and the reason why this is important is that models that
[01:17:46] this is important is that models that went through this type of Po actually
[01:17:49] went through this type of Po actually don't give you likelihoods of text that
[01:17:52] don't give you likelihoods of text that are meaningful cuz what you optimize
[01:17:54] are meaningful cuz what you optimize them to do is B basically just optimized
[01:17:56] them to do is B basically just optimized for generating the most likely thing not
[01:17:59] for generating the most likely thing not optimize for modeling like all the
[01:18:01] optimize for modeling like all the answers that humans might say another
[01:18:03] answers that humans might say another way of saying that is that there's
[01:18:04] way of saying that is that there's nothing that incentivizes here the model
[01:18:07] nothing that incentivizes here the model to not give a like a um a single
[01:18:10] to not give a like a um a single possible generation nothing here says
[01:18:13] possible generation nothing here says it's good if you have some distribution
[01:18:15] it's good if you have some distribution with some
[01:18:16] with some entropy um okay if you haven't followed
[01:18:19] entropy um okay if you haven't followed it's not that important but just good to
[01:18:21] it's not that important but just good to knowe great so PO is exact what chat GPT
[01:18:26] knowe great so PO is exact what chat GPT did originally so here's the on the blog
[01:18:29] did originally so here's the on the blog post or what they have is step one do
[01:18:32] post or what they have is step one do supervise fine training which now you
[01:18:34] supervise fine training which now you all know about step two train a reward
[01:18:36] all know about step two train a reward model on human preferences step three do
[01:18:39] model on human preferences step three do po multiple steps which is where you see
[01:18:42] po multiple steps which is where you see this this blue arrow so you continue you
[01:18:44] this this blue arrow so you continue you train the model once with po you collect
[01:18:46] train the model once with po you collect new data you continue uh and that's why
[01:18:48] new data you continue uh and that's why and that's exactly what Chad GPT did uh
[01:18:51] and that's exactly what Chad GPT did uh that was a big breakthrough between gpt3
[01:18:54] that was a big breakthrough between gpt3 and Chad GPT
[01:18:55] and Chad GPT one thing to note is that uh P has many
[01:18:58] one thing to note is that uh P has many challenges reinforcement learning is
[01:19:00] challenges reinforcement learning is something that's super nice
[01:19:02] something that's super nice theoretically in practice anyone who
[01:19:03] theoretically in practice anyone who ever worked with reinforcement learning
[01:19:05] ever worked with reinforcement learning knows it's such a mess uh there's a lot
[01:19:07] knows it's such a mess uh there's a lot of things like roll outs out of Loops
[01:19:09] of things like roll outs out of Loops clipping so many complications um so
[01:19:12] clipping so many complications um so it's messy this is the idealized PO used
[01:19:15] it's messy this is the idealized PO used for LM settings so that's already much
[01:19:17] for LM settings so that's already much more complicated than this expectation
[01:19:18] more complicated than this expectation we saw before and in practice it's
[01:19:20] we saw before and in practice it's actually much more complicated so we
[01:19:21] actually much more complicated so we have one implementation of it that we
[01:19:23] have one implementation of it that we had to do and I'm not going to go
[01:19:25] had to do and I'm not going to go through it but basically you have like
[01:19:26] through it but basically you have like so much stuff that you have to think
[01:19:28] so much stuff that you have to think about when you implement that type of of
[01:19:31] about when you implement that type of of uh po algorithm so you have clipping
[01:19:32] uh po algorithm so you have clipping everywhere you have a lot of
[01:19:34] everywhere you have a lot of complexities and things are not well
[01:19:37] complexities and things are not well documented all this to say um that we're
[01:19:40] documented all this to say um that we're going to there was a new method that was
[01:19:42] going to there was a new method that was proposed uh also from Sanford one year
[01:19:44] proposed uh also from Sanford one year ago called DPO which is essentially a
[01:19:47] ago called DPO which is essentially a simplification of Po um and the way uh
[01:19:52] simplification of Po um and the way uh what they did or the idea that they have
[01:19:54] what they did or the idea that they have is that instead of using reinforcement
[01:19:56] is that instead of using reinforcement learning you can just maximize the
[01:19:57] learning you can just maximize the probability of generating the stuff that
[01:19:59] probability of generating the stuff that you like and minimizing the probability
[01:20:01] you like and minimizing the probability of the stuff that you don't like uh so
[01:20:03] of the stuff that you don't like uh so if you think about the human preference
[01:20:04] if you think about the human preference the red and green maximize uh green
[01:20:07] the red and green maximize uh green minimize red um so the loss is actually
[01:20:11] minimize red um so the loss is actually this one uh where what you see this is
[01:20:13] this one uh where what you see this is simply um some log of the model so this
[01:20:17] simply um some log of the model so this is the likelihood of a model generating
[01:20:19] is the likelihood of a model generating the things that the human preferred
[01:20:20] the things that the human preferred given the the inputs um and what you try
[01:20:25] given the the inputs um and what you try to do is basically
[01:20:26] to do is basically maximize uh the likelihood of generating
[01:20:30] maximize uh the likelihood of generating the things that you like minimize the
[01:20:31] the things that you like minimize the likelihood of the things that you don't
[01:20:32] likelihood of the things that you don't like um all the rest of the terms here
[01:20:36] like um all the rest of the terms here it's not too important it's actually
[01:20:38] it's not too important it's actually really not that complicated to
[01:20:39] really not that complicated to understand but at a high level it's
[01:20:41] understand but at a high level it's really just maximizing the things you
[01:20:43] really just maximizing the things you like minimizing the the rest um and one
[01:20:47] like minimizing the the rest um and one thing to note uh which I was going to
[01:20:49] thing to note uh which I was going to say just here is that actually all the
[01:20:51] say just here is that actually all the rest is chosen such that um the global
[01:20:54] rest is chosen such that um the global Minima of of Po and a global Minima of
[01:20:57] Minima of of Po and a global Minima of like this DPO under some assumptions are
[01:21:00] like this DPO under some assumptions are essentially equivalent so this is the
[01:21:02] essentially equivalent so this is the right thing to do mathematically I'm not
[01:21:05] right thing to do mathematically I'm not going to go through the derivations but
[01:21:07] going to go through the derivations but that's the right thing to do uh it's
[01:21:08] that's the right thing to do uh it's pretty different with Po in the sense
[01:21:10] pretty different with Po in the sense that now and with P what you had to do
[01:21:12] that now and with P what you had to do is collect the human preferences then
[01:21:14] is collect the human preferences then train a uh reward model with maximum
[01:21:16] train a uh reward model with maximum likelihood then use reinforcement
[01:21:17] likelihood then use reinforcement learning now all you do is basically
[01:21:19] learning now all you do is basically maximum likelihood much simpler yes I
[01:21:21] maximum likelihood much simpler yes I mean yeah so it seems like this is a
[01:21:23] mean yeah so it seems like this is a much simpler and B like what you just
[01:21:25] much simpler and B like what you just intuitively do if this why did they
[01:21:28] intuitively do if this why did they start with this reward model like what
[01:21:30] start with this reward model like what what led them doing that I think it's a
[01:21:32] what led them doing that I think it's a great question uh I don't really know
[01:21:35] great question uh I don't really know what I can tell you is that at open ey
[01:21:37] what I can tell you is that at open ey the people who did the um uh who did
[01:21:41] the people who did the um uh who did basically this PP uh sorry who did Chad
[01:21:43] basically this PP uh sorry who did Chad GPT initially are the ones who actually
[01:21:46] GPT initially are the ones who actually wrote Po and I think they were just like
[01:21:49] wrote Po and I think they were just like there are a lot of reinforcement
[01:21:50] there are a lot of reinforcement learning people and I think that for
[01:21:52] learning people and I think that for them it was very intuitive um so there's
[01:21:56] them it was very intuitive um so there's also some additional like potential
[01:21:58] also some additional like potential benefits for example I don't want to
[01:22:01] benefits for example I don't want to yeah for example if you use the reward
[01:22:02] yeah for example if you use the reward model uh the cool thing here with
[01:22:04] model uh the cool thing here with reinforcement learning is that you can
[01:22:05] reinforcement learning is that you can use unlabeled data with the reward model
[01:22:08] use unlabeled data with the reward model so here you can only use the label data
[01:22:10] so here you can only use the label data for doing DPO um for PP for po you first
[01:22:14] for doing DPO um for PP for po you first train your reward model and then you can
[01:22:16] train your reward model and then you can use unlabeled data uh where the reward
[01:22:19] use unlabeled data uh where the reward model will basically label this
[01:22:20] model will basically label this unlabeled data so there there's
[01:22:22] unlabeled data so there there's additional kind of potential uh
[01:22:25] additional kind of potential uh there could be potential improvements in
[01:22:27] there could be potential improvements in practice it happens at down and on and I
[01:22:30] practice it happens at down and on and I think just that a lot of people in this
[01:22:32] think just that a lot of people in this team were reinforcement learning experts
[01:22:34] team were reinforcement learning experts including uh the main author of Po John
[01:22:38] including uh the main author of Po John hman um so much simpler in poo and is
[01:22:41] hman um so much simpler in poo and is basically performs as well uh so now
[01:22:44] basically performs as well uh so now this is the standard uh thing that
[01:22:45] this is the standard uh thing that people use at least in the open source
[01:22:47] people use at least in the open source Community I believe it's actually the
[01:22:49] Community I believe it's actually the standard also in in Industry so that's
[01:22:53] standard also in in Industry so that's called DPO gains
[01:22:55] called DPO gains um so those are all the papers on the
[01:22:57] um so those are all the papers on the left here this is on a summarization
[01:22:59] left here this is on a summarization task you see all I want to show you is
[01:23:02] task you see all I want to show you is that basically the pre-train models uh
[01:23:04] that basically the pre-train models uh were okay and they improve with scale if
[01:23:06] were okay and they improve with scale if you do supervised fine tuning you
[01:23:07] you do supervised fine tuning you improve them a little bit more if you do
[01:23:09] improve them a little bit more if you do po or something with all HF with human
[01:23:12] po or something with all HF with human feedback you get performance that are as
[01:23:15] feedback you get performance that are as often times depending on a benchmark
[01:23:17] often times depending on a benchmark even better than uh humans so this is
[01:23:19] even better than uh humans so this is the human uh reference summaries same
[01:23:22] the human uh reference summaries same thing this is on a uh on a paper that we
[01:23:24] thing this is on a uh on a paper that we have Alpaca Farm
[01:23:25] have Alpaca Farm where we see uh the evaluation here is
[01:23:27] where we see uh the evaluation here is not too important but basically you see
[01:23:28] not too important but basically you see pre-train model you jump to sft and then
[01:23:32] pre-train model you jump to sft and then you jump to PPO and popo have the exact
[01:23:34] you jump to PPO and popo have the exact same
[01:23:36] same performance so basically all HF helps
[01:23:39] performance so basically all HF helps that's kind of the conclusion and DPO is
[01:23:41] that's kind of the conclusion and DPO is simple uh data uh the way that you
[01:23:44] simple uh data uh the way that you collect that type of data um first idea
[01:23:48] collect that type of data um first idea is just use humans as we already talked
[01:23:50] is just use humans as we already talked about uh guidelines are very complicated
[01:23:53] about uh guidelines are very complicated for what humans should be labeling and
[01:23:54] for what humans should be labeling and and it's really not that easy and
[01:23:56] and it's really not that easy and actually if you ever do some of the
[01:23:57] actually if you ever do some of the labeling you will see that it's
[01:24:00] labeling you will see that it's extremely complicated like if I zoom in
[01:24:02] extremely complicated like if I zoom in to this uh here I have a question tell
[01:24:06] to this uh here I have a question tell tell me about self-driving cars and you
[01:24:08] tell me about self-driving cars and you read both self-driving cars are vehicles
[01:24:10] read both self-driving cars are vehicles that are capable of detecting their
[01:24:11] that are capable of detecting their surroundings blah blah blah self-driving
[01:24:13] surroundings blah blah blah self-driving cars are cars that are equipped with
[01:24:14] cars are cars that are equipped with sensors blah blah blah to navigate
[01:24:15] sensors blah blah blah to navigate without the need for a driver I mean
[01:24:17] without the need for a driver I mean both seem okay like which one is better
[01:24:19] both seem okay like which one is better it's actually hard to say at a glance um
[01:24:22] it's actually hard to say at a glance um and as a result uh the problem with
[01:24:24] and as a result uh the problem with humans is that you will start optimizing
[01:24:27] humans is that you will start optimizing a lot of like high level features for
[01:24:29] a lot of like high level features for example the second one is longer I can
[01:24:31] example the second one is longer I can guarantee you that most humans will
[01:24:32] guarantee you that most humans will choose second one even though I mean
[01:24:35] choose second one even though I mean maybe the first one is better I don't
[01:24:36] maybe the first one is better I don't know I haven't read it carefully so
[01:24:39] know I haven't read it carefully so challenges with humans first slow and
[01:24:41] challenges with humans first slow and expensive uh second as I just mentioned
[01:24:45] expensive uh second as I just mentioned it's hard to focus on things that matter
[01:24:47] it's hard to focus on things that matter like correctness and people uh usually
[01:24:49] like correctness and people uh usually look at things that don't matter as much
[01:24:51] look at things that don't matter as much like the form like length uh and as a
[01:24:54] like the form like length uh and as a result so what I show here is that uh
[01:24:56] result so what I show here is that uh when you do lhf the more you do of lhf
[01:24:58] when you do lhf the more you do of lhf the longer the output of the of the
[01:25:00] the longer the output of the of the models become so if you've ever been
[01:25:02] models become so if you've ever been annoyed at chat GPT answering you super
[01:25:05] annoyed at chat GPT answering you super long sentences this is because of all
[01:25:07] long sentences this is because of all rhf um annotator distribution shift uh
[01:25:12] rhf um annotator distribution shift uh like the distribution of annotators that
[01:25:13] like the distribution of annotators that you use matters a lot and you have to
[01:25:15] you use matters a lot and you have to think like what is what is even the
[01:25:17] think like what is what is even the humans that we want to represent in
[01:25:19] humans that we want to represent in these models uh now the question is like
[01:25:21] these models uh now the question is like crowdsourcing ethics uh like usually
[01:25:24] crowdsourcing ethics uh like usually these basically a lot of the the
[01:25:26] these basically a lot of the the labeling that is done um like the people
[01:25:29] labeling that is done um like the people who do them are not paid well and they
[01:25:31] who do them are not paid well and they have to go through a lot of toxic data
[01:25:33] have to go through a lot of toxic data uh because you basically want the model
[01:25:35] uh because you basically want the model to avoid saying the toxic data um so
[01:25:38] to avoid saying the toxic data um so crowdsourcing ethics
[01:25:39] crowdsourcing ethics too so many challenges with human data
[01:25:43] too so many challenges with human data um so what we did also last year is
[01:25:46] um so what we did also last year is again the same thing as alpaca just the
[01:25:47] again the same thing as alpaca just the idea of like oh well they're challenges
[01:25:49] idea of like oh well they're challenges with humans maybe we can just replace
[01:25:50] with humans maybe we can just replace them with llms uh so what we did is
[01:25:53] them with llms uh so what we did is simply replace
[01:25:55] simply replace um oh I see that I'm just realizing that
[01:25:58] um oh I see that I'm just realizing that the slides are not sented anyways uh you
[01:26:00] the slides are not sented anyways uh you replace a human preference with LM
[01:26:02] replace a human preference with LM preferences uh so here on this uh figure
[01:26:05] preferences uh so here on this uh figure you see on the xaxis the price that we
[01:26:07] you see on the xaxis the price that we paid uh for collecting human data it's
[01:26:09] paid uh for collecting human data it's around
[01:26:10] around $300 for 1,000 examples and this is on
[01:26:13] $300 for 1,000 examples and this is on mechanical turkers which are usually
[01:26:16] mechanical turkers which are usually like cheaper than than maybe some of the
[01:26:18] like cheaper than than maybe some of the other um companies that you could go
[01:26:20] other um companies that you could go through and on the Y AIS it's basically
[01:26:23] through and on the Y AIS it's basically the agreement with uh other humans with
[01:26:26] the agreement with uh other humans with the mode of other humans and what you
[01:26:27] the mode of other humans and what you see is that actually as I told you
[01:26:29] see is that actually as I told you before labeling is really complicated
[01:26:31] before labeling is really complicated humans agree with themselves only around
[01:26:33] humans agree with themselves only around 66% of the time on a binary Tas and it's
[01:26:37] 66% of the time on a binary Tas and it's not that the humans are not good here
[01:26:39] not that the humans are not good here because uh we were five main authors on
[01:26:41] because uh we were five main authors on this paper we tried to label this data
[01:26:43] this paper we tried to label this data ourselves and we only had like say 67 or
[01:26:46] ourselves and we only had like say 67 or 68% accuracy even though we talk like we
[01:26:49] 68% accuracy even though we talk like we talk for like 3 hours of how we should
[01:26:50] talk for like 3 hours of how we should be doing labeling really it's
[01:26:52] be doing labeling really it's complicated it's not an easy task um and
[01:26:55] complicated it's not an easy task um and here I just showed many different models
[01:26:56] here I just showed many different models and um basically you see that models are
[01:26:59] and um basically you see that models are much cheaper and they can actually get
[01:27:00] much cheaper and they can actually get higher agreement with the mode of humans
[01:27:03] higher agreement with the mode of humans than human humans themselves and the
[01:27:05] than human humans themselves and the reason why is because humans have a lot
[01:27:06] reason why is because humans have a lot of varant models have no varant so they
[01:27:08] of varant models have no varant so they might be a little bit more biased but
[01:27:10] might be a little bit more biased but have less virence uh so it works
[01:27:12] have less virence uh so it works surprisingly well and now it's kind of
[01:27:14] surprisingly well and now it's kind of the standard in open uh Source Community
[01:27:17] the standard in open uh Source Community I think even in Industry a lot of people
[01:27:19] I think even in Industry a lot of people use both humans and llms for improving
[01:27:22] use both humans and llms for improving uh the colle collection of allf data
[01:27:25] uh the colle collection of allf data um and this is like this is the paper
[01:27:27] um and this is like this is the paper from last year but honestly now it's
[01:27:28] from last year but honestly now it's more like that llms would be around this
[01:27:31] more like that llms would be around this agreement and this cost so around I
[01:27:33] agreement and this cost so around I would say 50x cheaper than humans and
[01:27:35] would say 50x cheaper than humans and better agreement with human than humans
[01:27:38] better agreement with human than humans themselves okay so that gets us to
[01:27:42] themselves okay so that gets us to evaluation of post
[01:27:43] evaluation of post training um that goes back to your
[01:27:46] training um that goes back to your initial question at the beginning of the
[01:27:47] initial question at the beginning of the lecture how do you evaluate something
[01:27:49] lecture how do you evaluate something like chpt uh the answers that chpt could
[01:27:52] like chpt uh the answers that chpt could give are basically unbounded and it's
[01:27:55] give are basically unbounded and it's not that there one right answer there
[01:27:56] not that there one right answer there are many answers that are just as good
[01:27:59] are many answers that are just as good um so there are many challenges one you
[01:28:01] um so there are many challenges one you can't use validation loss because one
[01:28:05] can't use validation loss because one method might use po the other one might
[01:28:07] method might use po the other one might use DPO validation loss is not
[01:28:08] use DPO validation loss is not comparable second you can't use Cal uh
[01:28:11] comparable second you can't use Cal uh sorry perplexity that's the thing I told
[01:28:13] sorry perplexity that's the thing I told you before these models uh are not
[01:28:15] you before these models uh are not calibrated they don't give distributions
[01:28:18] calibrated they don't give distributions they they just optimize for one thing so
[01:28:20] they they just optimize for one thing so you can't use perplexity for actually
[01:28:21] you can't use perplexity for actually evaluating uh these type of models once
[01:28:23] evaluating uh these type of models once they're aligned sorry one Z lined third
[01:28:28] they're aligned sorry one Z lined third uh there's a large diversity of
[01:28:29] uh there's a large diversity of questions that human might ask to these
[01:28:31] questions that human might ask to these models generation open QA like some
[01:28:33] models generation open QA like some question answering some summarization
[01:28:35] question answering some summarization and all of these things so there's so
[01:28:36] and all of these things so there's so many things you have to cover um then
[01:28:39] many things you have to cover um then the tasks are really open-ended so it's
[01:28:41] the tasks are really open-ended so it's very hard to automate so that's what you
[01:28:43] very hard to automate so that's what you were alluding to before so the idea uh
[01:28:47] were alluding to before so the idea uh is that instead of trying to come up
[01:28:49] is that instead of trying to come up with really easily automated uh
[01:28:51] with really easily automated uh benchmarks uh it's just we're going to
[01:28:53] benchmarks uh it's just we're going to ask questions that that users actually
[01:28:55] ask questions that that users actually ask to these models in practice and
[01:28:57] ask to these models in practice and we're just going to ask annotators to
[01:28:59] we're just going to ask annotators to say between these two models which one
[01:29:01] say between these two models which one is better like what's the what's the
[01:29:02] is better like what's the what's the better output so basically do exact same
[01:29:04] better output so basically do exact same thing as um basically the data from rhf
[01:29:09] thing as um basically the data from rhf but you use it now for evaluation yes
[01:29:11] but you use it now for evaluation yes I'm not sure I understand what you mean
[01:29:12] I'm not sure I understand what you mean by like can't use perplexity and not
[01:29:14] by like can't use perplexity and not calibrated right like LM is still doing
[01:29:16] calibrated right like LM is still doing like next token
[01:29:18] like next token prediction so I can't so think about um
[01:29:23] prediction so I can't so think about um the optim solution after doing PO is
[01:29:26] the optim solution after doing PO is basically one model that gives you uh
[01:29:29] basically one model that gives you uh essentially a Delta um like basically
[01:29:32] essentially a Delta um like basically says that there's only one sentence that
[01:29:34] says that there's only one sentence that is that could be generated for that
[01:29:36] is that could be generated for that question so now if you use it on
[01:29:38] question so now if you use it on something that is slightly semantically
[01:29:40] something that is slightly semantically differently different it would actually
[01:29:41] differently different it would actually give a likelihood of zero for that
[01:29:43] give a likelihood of zero for that answer so in reality it's not that
[01:29:46] answer so in reality it's not that extreme because as you say it's still a
[01:29:47] extreme because as you say it's still a distribution but I just shows you that
[01:29:49] distribution but I just shows you that there's a there's a fundamental issue
[01:29:50] there's a there's a fundamental issue with perplexity once these models are
[01:29:53] with perplexity once these models are not llms anymore they were not trained
[01:29:56] not llms anymore they were not trained at least with P they were not trained to
[01:29:58] at least with P they were not trained to to do maximum likelihood anymore they
[01:29:59] to do maximum likelihood anymore they were trained to be
[01:30:06] policies okay um so probably the most common or like the most um yeah the most
[01:30:09] common or like the most um yeah the most common Benchmark or the most trusted one
[01:30:11] common Benchmark or the most trusted one is what we call Chad uh sorry chatbot
[01:30:13] is what we call Chad uh sorry chatbot Arena uh which is basically go on
[01:30:15] Arena uh which is basically go on internet have random users on the
[01:30:17] internet have random users on the internet blindly talk with two chat Bots
[01:30:20] internet blindly talk with two chat Bots just ask many questions see the two
[01:30:22] just ask many questions see the two answers and rate which one is better and
[01:30:24] answers and rate which one is better and and you do that over hundred of
[01:30:26] and you do that over hundred of thousands of users and then you get uh
[01:30:28] thousands of users and then you get uh the actual preferences and you get
[01:30:29] the actual preferences and you get rankings of models uh so you can go
[01:30:32] rankings of models uh so you can go right now on chatbot Arena and actually
[01:30:34] right now on chatbot Arena and actually interact with these models um one
[01:30:37] interact with these models um one potential issue just to highlight is
[01:30:39] potential issue just to highlight is that while people who want to do these
[01:30:40] that while people who want to do these type of things are usually more like
[01:30:41] type of things are usually more like Tech driven um or like techsavvy uh so a
[01:30:45] Tech driven um or like techsavvy uh so a lot of the questions that you will ask
[01:30:46] lot of the questions that you will ask are more like Tech stuff discussing
[01:30:48] are more like Tech stuff discussing software errors inquiries about AI tools
[01:30:50] software errors inquiries about AI tools and all these things um so another issue
[01:30:54] and all these things um so another issue is cost and speed if you really want to
[01:30:56] is cost and speed if you really want to use something like this for development
[01:30:57] use something like this for development process um it will be too costly because
[01:31:00] process um it will be too costly because you would need to basically pay a lot of
[01:31:02] you would need to basically pay a lot of humans to do that so one simple idea is
[01:31:06] humans to do that so one simple idea is again as we said many times just use LM
[01:31:09] again as we said many times just use LM instead of humans uh you probably know
[01:31:11] instead of humans uh you probably know the drill at this point uh steps for
[01:31:14] the drill at this point uh steps for every instruction generate outputs by
[01:31:16] every instruction generate outputs by some baseline and the model that you
[01:31:18] some baseline and the model that you want to evaluate um so here you imagine
[01:31:20] want to evaluate um so here you imagine that I I'm comparing an answer from Chad
[01:31:23] that I I'm comparing an answer from Chad GPT and from
[01:31:25] GPT and from I'm just asking a model uh another model
[01:31:28] I'm just asking a model uh another model uh which one is better and I just
[01:31:31] uh which one is better and I just basically average that out uh yeah I
[01:31:33] basically average that out uh yeah I asked gp4 which one is better I average
[01:31:35] asked gp4 which one is better I average that out over my entire distribution
[01:31:37] that out over my entire distribution over my entire Benchmark or data set and
[01:31:40] over my entire Benchmark or data set and that gives me a RN rate so RN
[01:31:42] that gives me a RN rate so RN probability for one model compared to
[01:31:44] probability for one model compared to another one and now you can rank models
[01:31:47] another one and now you can rank models uh and this is the Alpa eval uh
[01:31:50] uh and this is the Alpa eval uh leaderboard so the benefits of this is
[01:31:52] leaderboard so the benefits of this is that actually we show we get 98%
[01:31:55] that actually we show we get 98% correlation with Chad B Arena so very
[01:31:56] correlation with Chad B Arena so very high correlation with humans um so this
[01:32:00] high correlation with humans um so this is yeah comparison with correlation with
[01:32:02] is yeah comparison with correlation with other benchmarks and it takes less than
[01:32:04] other benchmarks and it takes less than three minutes and less than $10 to run
[01:32:06] three minutes and less than $10 to run so it's pretty cheap um there are
[01:32:08] so it's pretty cheap um there are downsides though uh one of them is purus
[01:32:10] downsides though uh one of them is purus correlation um so as we already saw
[01:32:13] correlation um so as we already saw before LMS prefer this is one SP
[01:32:16] before LMS prefer this is one SP correlation not many I'll just talk
[01:32:17] correlation not many I'll just talk about one LMS prefer longer outputs
[01:32:19] about one LMS prefer longer outputs actually humans also prefer longer
[01:32:20] actually humans also prefer longer outputs but the problem or the issue
[01:32:22] outputs but the problem or the issue once you use llms is that once there
[01:32:24] once you use llms is that once there bias you will continue optimizing that
[01:32:26] bias you will continue optimizing that humans at some point I can guarantee you
[01:32:28] humans at some point I can guarantee you if I ask a simple question and you give
[01:32:29] if I ask a simple question and you give me five pages of answers I'll be like no
[01:32:31] me five pages of answers I'll be like no I don't like that answer but LMS if they
[01:32:33] I don't like that answer but LMS if they have this bius and they were trained for
[01:32:35] have this bius and they were trained for that they will continue preferring
[01:32:36] that they will continue preferring longer outputs so uh here we see um the
[01:32:41] longer outputs so uh here we see um the the preference just showing that like
[01:32:43] the preference just showing that like humans and models prefer longer outputs
[01:32:46] humans and models prefer longer outputs um and here is another view of the
[01:32:48] um and here is another view of the initial apaka eval data uh Benchmark
[01:32:51] initial apaka eval data uh Benchmark where when we asked um when we we rank
[01:32:54] where when we asked um when we we rank gp4 when we look at the Run rate of gp4
[01:32:57] gp4 when we look at the Run rate of gp4 versus actually uh gp4 itself if we com
[01:33:00] versus actually uh gp4 itself if we com if we use the standard GPT 4 it gets 50%
[01:33:03] if we use the standard GPT 4 it gets 50% kind of by definition because we're
[01:33:04] kind of by definition because we're comparing GPT 4 versus gp4 but if we ask
[01:33:07] comparing GPT 4 versus gp4 but if we ask a gbd4 to be slightly more verose so we
[01:33:09] a gbd4 to be slightly more verose so we just say in the prompt be Vos in your
[01:33:12] just say in the prompt be Vos in your answers then it gets a r rate of
[01:33:14] answers then it gets a r rate of 64.4% so really there's a huge variance
[01:33:17] 64.4% so really there's a huge variance and if we ask it to be concise it gets
[01:33:18] and if we ask it to be concise it gets 20% so there's a huge variance depending
[01:33:21] 20% so there's a huge variance depending on um whether you ask it to be concise
[01:33:23] on um whether you ask it to be concise of
[01:33:24] of that's very annoying um so one possible
[01:33:28] that's very annoying um so one possible solution which is what we did is uh just
[01:33:30] solution which is what we did is uh just use some regression analysis I'm not
[01:33:32] use some regression analysis I'm not going to go into details but basically
[01:33:33] going to go into details but basically use Cal inference tools to control for
[01:33:35] use Cal inference tools to control for length and right now uh actually length
[01:33:38] length and right now uh actually length matters much less so if you ask it to be
[01:33:40] matters much less so if you ask it to be veros we still get some gains but much
[01:33:44] veros we still get some gains but much less great so that's all about post
[01:33:46] less great so that's all about post training and now for the next eight
[01:33:48] training and now for the next eight minutes I might talk about systems or
[01:33:50] minutes I might talk about systems or just answer questions yes can you um go
[01:33:54] just answer questions yes can you um go back to your post training in terms of
[01:33:56] back to your post training in terms of post training how did we tune those
[01:33:59] post training how did we tune those parameters using the small body of
[01:34:02] parameters using the small body of fine-tuning data and have such big
[01:34:04] fine-tuning data and have such big effect on the model you mentioned
[01:34:06] effect on the model you mentioned earlier that there's a different set of
[01:34:08] earlier that there's a different set of hyperparameters are we changing just
[01:34:10] hyperparameters are we changing just some of the weights the later weights or
[01:34:12] some of the weights the later weights or all the weights what's actually
[01:34:13] all the weights what's actually happening yeah uh yeah I I kind of
[01:34:16] happening yeah uh yeah I I kind of skimmed through all of this you change
[01:34:17] skimmed through all of this you change all the weights actually um industry
[01:34:19] all the weights actually um industry would change all the weights in open
[01:34:21] would change all the weights in open source land you might have heard of
[01:34:23] source land you might have heard of Laura which is going to change basically
[01:34:26] Laura which is going to change basically only some of the weights or it actually
[01:34:28] only some of the weights or it actually to be more specific it's going to add
[01:34:30] to be more specific it's going to add some differences to the output of every
[01:34:32] some differences to the output of every of every layer but but in Industry
[01:34:34] of every layer but but in Industry you're going to just fine tune all the
[01:34:36] you're going to just fine tune all the weights um and also to say something
[01:34:40] weights um and also to say something else about the data actually the SL St
[01:34:41] else about the data actually the SL St all HF you usually going to collect uh a
[01:34:44] all HF you usually going to collect uh a lot more data than with sft so if fft is
[01:34:47] lot more data than with sft so if fft is like 5,000 10,000 maybe 50,000 with rhf
[01:34:51] like 5,000 10,000 maybe 50,000 with rhf I think you're going to be more around
[01:34:53] I think you're going to be more around like the 1 million
[01:34:54] like the 1 million uh order of magnitude it's still much
[01:34:56] uh order of magnitude it's still much less than pre-training though yeah
[01:34:58] less than pre-training though yeah because pre-training is 15 trillion
[01:35:00] because pre-training is 15 trillion tokens I mean this is like that's not
[01:35:02] tokens I mean this is like that's not even a drop and yet you influence the
[01:35:04] even a drop and yet you influence the weight a lot so because you do it I mean
[01:35:06] weight a lot so because you do it I mean you have to think that how you do it is
[01:35:08] you have to think that how you do it is you use um I mean as I said the learning
[01:35:12] you use um I mean as I said the learning rate that you're going to use is going
[01:35:13] rate that you're going to use is going to be different but also you only do
[01:35:16] to be different but also you only do that so just imagine if I train even if
[01:35:18] that so just imagine if I train even if I train on one sentence but over and
[01:35:21] I train on one sentence but over and over again all at some point my model
[01:35:23] over again all at some point my model will only that sentence even if uh it
[01:35:26] will only that sentence even if uh it was just one sentence instead of the 15
[01:35:28] was just one sentence instead of the 15 trillion tokens so if you use a large
[01:35:30] trillion tokens so if you use a large enough learning rate and for enough time
[01:35:33] enough learning rate and for enough time you will basically overfit that sentence
[01:35:35] you will basically overfit that sentence so the the the key thing to to remember
[01:35:37] so the the the key thing to to remember is that um the data is not I it's not as
[01:35:40] is that um the data is not I it's not as if you mix some posttraining data and
[01:35:43] if you mix some posttraining data and some pre-training data you do
[01:35:44] some pre-training data you do pre-training and then you just start
[01:35:46] pre-training and then you just start fine-tuning only on the post trining so
[01:35:48] fine-tuning only on the post trining so another way maybe another perspective is
[01:35:51] another way maybe another perspective is that the post the pre-training is just
[01:35:52] that the post the pre-training is just the initialization of your model
[01:35:54] the initialization of your model and once you view it that way that this
[01:35:56] and once you view it that way that this is just initialization of Weights then
[01:35:58] is just initialization of Weights then there's nothing special like you don't
[01:36:00] there's nothing special like you don't need to remember that you train a lot of
[01:36:02] need to remember that you train a lot of data before the only thing that matters
[01:36:03] data before the only thing that matters is that you had an initialization and
[01:36:05] is that you had an initialization and now I actually train a model so maybe
[01:36:07] now I actually train a model so maybe think about it that way like there's a
[01:36:09] think about it that way like there's a there's a mark of property in some way
[01:36:11] there's a mark of property in some way just like you had your weights this is
[01:36:12] just like you had your weights this is my initialization now I'm training that
[01:36:14] my initialization now I'm training that one does that kind of answer your
[01:36:15] one does that kind of answer your question kind of but you said something
[01:36:19] question kind of but you said something just now about it's almost the
[01:36:21] just now about it's almost the equivalence of just rerunning the find
[01:36:24] equivalence of just rerunning the find tuning data many times is it actually is
[01:36:26] tuning data many times is it actually is that what actually happens in order to
[01:36:29] that what actually happens in order to give so much more preference
[01:36:33] give so much more preference um you might I actually don't know right
[01:36:36] um you might I actually don't know right now how they do it in Industry when we
[01:36:38] now how they do it in Industry when we did alpaca we had to do three box so you
[01:36:40] did alpaca we had to do three box so you did run it three times to it
[01:36:44] did run it three times to it um but I mean even the number of times
[01:36:46] um but I mean even the number of times that you run it through it's actually
[01:36:48] that you run it through it's actually not important the only thing like the
[01:36:50] not important the only thing like the only thing is the is kind of the
[01:36:52] only thing is the is kind of the effective learning rate that what
[01:36:53] effective learning rate that what matters
[01:36:54] matters um so
[01:36:56] um so yeah
[01:36:58] yeah great so I think I have five minutes
[01:37:05] [Music] right okay I might try to give a high
[01:37:11] right okay I might try to give a high level Overview at least from one of the
[01:37:13] level Overview at least from one of the systems trick systems as we said uh for
[01:37:18] systems trick systems as we said uh for everyone Bott neck is a sorry compute is
[01:37:20] everyone Bott neck is a sorry compute is the huge bottleneck uh one question you
[01:37:22] the huge bottleneck uh one question you might ask is why not buy more gpus uh
[01:37:25] might ask is why not buy more gpus uh gpus are expensive but also are scarce
[01:37:27] gpus are expensive but also are scarce even if you have $10 million right now
[01:37:29] even if you have $10 million right now you cannot buy the best gpus um
[01:37:32] you cannot buy the best gpus um there's oh yeah there's also some
[01:37:34] there's oh yeah there's also some physical limitations when you have when
[01:37:36] physical limitations when you have when you have multiple gpus you have to
[01:37:38] you have multiple gpus you have to communicate between them that takes time
[01:37:41] communicate between them that takes time um so just buying more gpus is not that
[01:37:43] um so just buying more gpus is not that easy um so it's really important to
[01:37:45] easy um so it's really important to think about how do you allocate
[01:37:46] think about how do you allocate resources and how do you optimize your
[01:37:48] resources and how do you optimize your pipeline so system 101 on gpus I'm sorry
[01:37:52] pipeline so system 101 on gpus I'm sorry I'm going slightly faster I hope for
[01:37:54] I'm going slightly faster I hope for that some of you at least can follow uh
[01:37:56] that some of you at least can follow uh gpus are basically optimized for
[01:37:57] gpus are basically optimized for throughput CPUs are optimized uh for
[01:38:01] throughput CPUs are optimized uh for latency so gpus the way you have to
[01:38:03] latency so gpus the way you have to think about it is that there's one Comm
[01:38:05] think about it is that there's one Comm there's one command that is run on many
[01:38:07] there's one command that is run on many many Calles at the same time on
[01:38:09] many Calles at the same time on different type of data um so this is how
[01:38:12] different type of data um so this is how you see a GPU you see there are many
[01:38:14] you see a GPU you see there are many different CES we call them streaming
[01:38:16] different CES we call them streaming multiprocessors which is very different
[01:38:18] multiprocessors which is very different than the usual CPU architecture so just
[01:38:21] than the usual CPU architecture so just think High throughput paralyzation for
[01:38:24] think High throughput paralyzation for gpus uh gpus are optimized for fast
[01:38:26] gpus uh gpus are optimized for fast matrix multiplication so every time you
[01:38:29] matrix multiplication so every time you will do uh you will do something on GPU
[01:38:31] will do uh you will do something on GPU if you can do it with a a matrix
[01:38:33] if you can do it with a a matrix multiplication it's going to be 10 times
[01:38:34] multiplication it's going to be 10 times faster than with anything else uh that
[01:38:37] faster than with anything else uh that is a little bit annoying because it
[01:38:38] is a little bit annoying because it means that we're kind of uh bottlenecked
[01:38:41] means that we're kind of uh bottlenecked to doing anything with Matrix
[01:38:43] to doing anything with Matrix multiplications um another thing to note
[01:38:45] multiplications um another thing to note with gpus is that compute has been
[01:38:48] with gpus is that compute has been improving faster than memory and
[01:38:49] improving faster than memory and communication so right now gpus usually
[01:38:54] communication so right now gpus usually are hard to keep uh like the data that
[01:38:57] are hard to keep uh like the data that you send that send to gpus is actually
[01:38:59] you send that send to gpus is actually hard to keep up with the processess so
[01:39:01] hard to keep up with the processess so most of your gpus are actually going to
[01:39:02] most of your gpus are actually going to be idle if you just run normal code if
[01:39:05] be idle if you just run normal code if you don't optimize your code so
[01:39:07] you don't optimize your code so communication and this will continue
[01:39:09] communication and this will continue over time another thing to know about
[01:39:12] over time another thing to know about gpus is that there's a memory hierarchy
[01:39:14] gpus is that there's a memory hierarchy this is the same thing actually with
[01:39:15] this is the same thing actually with CPUs but basically the closer you are to
[01:39:17] CPUs but basically the closer you are to your cuse the less memory there is but
[01:39:19] your cuse the less memory there is but the faster things run if you're further
[01:39:22] the faster things run if you're further more memory slower
[01:39:24] more memory slower um okay I'm going to skip that okay
[01:39:27] um okay I'm going to skip that okay actually I'm going to say it I told you
[01:39:28] actually I'm going to say it I told you about this uh the fact of communication
[01:39:31] about this uh the fact of communication uh the metric that people usually look
[01:39:32] uh the metric that people usually look at is model flop utilization so what is
[01:39:35] at is model flop utilization so what is the theoretical maximum that GPU could
[01:39:37] the theoretical maximum that GPU could run at no more flops that you could use
[01:39:39] run at no more flops that you could use per second divide sorry the number of OB
[01:39:42] per second divide sorry the number of OB observed through put divided by this
[01:39:44] observed through put divided by this theoretical um maximum and in general if
[01:39:47] theoretical um maximum and in general if you reach 50% you're very happy like
[01:39:50] you reach 50% you're very happy like Facebook I looked at Lama was at 45 or
[01:39:52] Facebook I looked at Lama was at 45 or something like this so that that means
[01:39:54] something like this so that that means that data doesn't come fast enough even
[01:39:56] that data doesn't come fast enough even for these big
[01:39:58] for these big companies so one simple trick and that
[01:40:01] companies so one simple trick and that might be the only one I'm going to tell
[01:40:02] might be the only one I'm going to tell you about is low Precision one simple
[01:40:05] you about is low Precision one simple idea is that well if I'm going to put my
[01:40:08] idea is that well if I'm going to put my floats in lower Precision then there's
[01:40:10] floats in lower Precision then there's going to be fewer bits that I have to
[01:40:11] going to be fewer bits that I have to send to my gpus if there's fewer bits
[01:40:13] send to my gpus if there's fewer bits it's faster communication lower memory
[01:40:15] it's faster communication lower memory consumption things are going to go
[01:40:17] consumption things are going to go faster uh and for deep learning it just
[01:40:19] faster uh and for deep learning it just happens that de decimal is not that
[01:40:22] happens that de decimal is not that important uh so so when you do matrix
[01:40:25] important uh so so when you do matrix multiplication when you do like for
[01:40:26] multiplication when you do like for example SGD there's already so much
[01:40:28] example SGD there's already so much noise that if you update something by
[01:40:30] noise that if you update something by 0.01 or
[01:40:32] 0.01 or 0.015 who cares uh so basically instead
[01:40:35] 0.015 who cares uh so basically instead of using uh 32 bits per float which is
[01:40:39] of using uh 32 bits per float which is um what people used to use or 64 for
[01:40:41] um what people used to use or 64 for example which is what you would use in
[01:40:43] example which is what you would use in other domains you use 16 bits uh for
[01:40:45] other domains you use 16 bits uh for matrix multiplication so for every float
[01:40:47] matrix multiplication so for every float you use 16 bits um and for training you
[01:40:51] you use 16 bits um and for training you have this type of like uh what we call
[01:40:53] have this type of like uh what we call aut atic mix Precision which is that uh
[01:40:55] aut atic mix Precision which is that uh some of the things are in 32 bits others
[01:40:57] some of the things are in 32 bits others are in 60 bit in 16 bits um generally
[01:41:01] are in 60 bit in 16 bits um generally the way you should be thinking about it
[01:41:02] the way you should be thinking about it is that your weights are stored of your
[01:41:05] is that your weights are stored of your model are stored in 32 bits um but just
[01:41:08] model are stored in 32 bits um but just before the computation you put
[01:41:09] before the computation you put everything in 16 16 bits like this you
[01:41:11] everything in 16 16 bits like this you do computation super fast and at the end
[01:41:14] do computation super fast and at the end you update your weights in 32 Bits And
[01:41:16] you update your weights in 32 Bits And the reason why you do all the updates in
[01:41:18] the reason why you do all the updates in 32 bits it's just think that if your
[01:41:20] 32 bits it's just think that if your learning rate for example is very small
[01:41:21] learning rate for example is very small you still want to be able to like make a
[01:41:24] you still want to be able to like make a difference in your weights uh so all the
[01:41:26] difference in your weights uh so all the computation is done in 16 bits but the
[01:41:29] computation is done in 16 bits but the weights are actually stored in 32 bits
[01:41:31] weights are actually stored in 32 bits so that's like the standard way that
[01:41:32] so that's like the standard way that people are doing it um okay I'll
[01:41:36] people are doing it um okay I'll actually talk just about this and then
[01:41:37] actually talk just about this and then I'll skip all the rest operator Fusion
[01:41:39] I'll skip all the rest operator Fusion because I think this is actually pretty
[01:41:40] because I think this is actually pretty cool as I just said communication is
[01:41:42] cool as I just said communication is very slow and actually every time you
[01:41:44] very slow and actually every time you use a pie torch line it basically moves
[01:41:47] use a pie torch line it basically moves variable to Global memory of your GPU so
[01:41:49] variable to Global memory of your GPU so when you have something like this x do
[01:41:52] when you have something like this x do cosine uh equal X1 and then you do X1 do
[01:41:56] cosine uh equal X1 and then you do X1 do cosine what is happening behind the
[01:41:58] cosine what is happening behind the scenes is that you take the X which is
[01:42:00] scenes is that you take the X which is data you ship it to your um to your
[01:42:02] data you ship it to your um to your actual processes of your gpus you apply
[01:42:04] actual processes of your gpus you apply the coign you ship it back to the main
[01:42:06] the coign you ship it back to the main memory of your GPU and then you see the
[01:42:09] memory of your GPU and then you see the next sign you ship it back to the
[01:42:10] next sign you ship it back to the computer to the GPU processor you apply
[01:42:13] computer to the GPU processor you apply another cosign and you ship it back
[01:42:15] another cosign and you ship it back again um so another way to see that is
[01:42:17] again um so another way to see that is that you go from your Dam which is your
[01:42:19] that you go from your Dam which is your Global memory in your GPU and you ship
[01:42:22] Global memory in your GPU and you ship it to compute you ship it back for every
[01:42:24] it to compute you ship it back for every line This is a naive way of doing it
[01:42:26] line This is a naive way of doing it this seems very wasteful um so the idea
[01:42:30] this seems very wasteful um so the idea simple idea of operative Fusion is just
[01:42:32] simple idea of operative Fusion is just communicate do all the computation ship
[01:42:35] communicate do all the computation ship it back once and this is exactly what
[01:42:38] it back once and this is exactly what fuse kernels are um so if you ever want
[01:42:40] fuse kernels are um so if you ever want to make your comp your computations in
[01:42:44] to make your comp your computations in pytorch much faster just apply torch.
[01:42:47] pytorch much faster just apply torch. compile on your model this is going to
[01:42:50] compile on your model this is going to make your model around two times faster
[01:42:52] make your model around two times faster and what it does is simply that it
[01:42:54] and what it does is simply that it rewrites your code uh your P like your
[01:42:57] rewrites your code uh your P like your py torch code basically in C++ in Cuda
[01:43:01] py torch code basically in C++ in Cuda uh to to do the communication only once
[01:43:04] uh to to do the communication only once then do all the operations then uh ship
[01:43:06] then do all the operations then uh ship it back okay I'm not going to have time
[01:43:09] it back okay I'm not going to have time to talk about tiling tiling is important
[01:43:12] to talk about tiling tiling is important paration paration is important um and
[01:43:16] paration paration is important um and mixture of experts mixture of experts is
[01:43:18] mixture of experts mixture of experts is important Outlook there are many things
[01:43:21] important Outlook there are many things we haven't T talked about we haven't
[01:43:24] we haven't T talked about we haven't talked about architectures we definitely
[01:43:25] talked about architectures we definitely haven't talked about inference um there
[01:43:28] haven't talked about inference um there are many other things that are important
[01:43:29] are many other things that are important with LMS what is the UI that you use I
[01:43:31] with LMS what is the UI that you use I mean arguably chat jpt the big novelty
[01:43:34] mean arguably chat jpt the big novelty was just have a simple UI to use it
[01:43:36] was just have a simple UI to use it multimodality what are all the misuses
[01:43:38] multimodality what are all the misuses you could have uh the fact that there
[01:43:40] you could have uh the fact that there might not be enough data on the internet
[01:43:41] might not be enough data on the internet to train all these models legality of
[01:43:43] to train all these models legality of data collection so many other things if
[01:43:45] data collection so many other things if you are interested in all these topics
[01:43:48] you are interested in all these topics uh I would suggest three classes cs224n
[01:43:51] uh I would suggest three classes cs224n is probably the one that touches the
[01:43:52] is probably the one that touches the least on uh LMS uh but it gives some
[01:43:56] least on uh LMS uh but it gives some background and historical context um of
[01:43:58] background and historical context um of all the LMS and gives kind of some
[01:44:00] all the LMS and gives kind of some adjacent material CS 324 I think it's
[01:44:03] adjacent material CS 324 I think it's called Uh I think it's just called large
[01:44:06] called Uh I think it's just called large language models uh more in-depth reading
[01:44:08] language models uh more in-depth reading and lectures on everything I talked
[01:44:10] and lectures on everything I talked about CS 336 which is large language
[01:44:13] about CS 336 which is large language model from scratch you actually build
[01:44:15] model from scratch you actually build your own llm uh it's an amazing class
[01:44:18] your own llm uh it's an amazing class also given by my two supervisors very
[01:44:21] also given by my two supervisors very heavy workload so be careful and um
[01:44:24] heavy workload so be careful and um great

All frames

Total: 12. Hero frames flagged with star.

• frames/frame_0001.jpg (t=00:00) frames/frame_0002.jpg (t=04:19)
• frames/frame_0003.jpg (t=13:15) frames/frame_0004.jpg (t=21:47)
• frames/frame_0005.jpg (t=38:53) frames/frame_0006.jpg (t=47:55)
• frames/frame_0007.jpg (t=56:34)
• frames/frame_0008.jpg (t=01:05:11) frames/frame_0009.jpg (t=01:22:36) frames/frame_0010.jpg (t=01:32:24) frames/frame_0011.jpg (t=01:39:58) frames/frame_0012.jpg (t=01:44:22)