NON_INT SOFTWARE & ML

Non_Int

  1. We Should Slow Down

    I regrettably did not sign the pacing the frontier petition because I was on vacation at the time it was circulating. But it is something I want to come out as categorically in favor of. I believe that we are on the precipice of an age of enormous technological progress packaged into an extremely short period of time. I fear that these rapid changes will strain our social infrastructure, which is already quite fragile. We should pull back on the throttle a bit to give ourselves a chance to adapt. What particularly worries me is that the two companies behind this technology are driven by market competition and a need to pay off their debts to justify their valuations. If AI is as transformative as they think it will be, we're going to rush headfirst into the singularity with rocket engines strapped to our backs.

  2. Vibe Coding

    I had a pretty incredible vibe coding experience with o3 today. As I’m sure many of you have also had recently – whether with o3, or Claude or Gemini. I was iterating on a problem with it over a couple of hours. I asked it to come up with an idea for a novel algorithm application for an advanced ML project. Then asking it to implement it. And write unit tests. I did static code analysis on the code it generated with the assistance of a separate instance of o3 before running anything.

  3. Mixture of Experts

    A Transformer is a stack of alternating Attention and MLP layers through which data embedded as high dimensional vectors is fed. A Mixture of Experts (MoE) Transformer substitutes the MLP layer for an “MoE Layer”. Let’s dive into what that means. The “MLP” is one of the oldest neural network architectures, consisting of two linear transformations. First, an embedding vector is expanded via the first transformation. Next, a non-linearity is applied to the expanded vector. Finally, it is contracted back to the original dimensionality.

  4. The Paradigm

    Over the past decade, some of the most remarkable AI breakthroughs—AlphaGo, AlphaStar, AlphaFold 1 , VPT, OpenAI Five, ChatGPT—have all shared a common thread: they start with large-scale data gathering (self-supervised or imitation learning, or SSL) and then use reinforcement learning to refine their performance toward a specific goal. This marriage of general knowledge acquisition and focused, reward-driven specialization has emerged as a the paradigm by which we can reliably train AI systems to excel at arbitrary tasks. I’d like to talk about how and why this works so well. 1 – AlphaFold 2 technically does not use RL; instead it uses distillation via rejection sampling, which has similar (if less adaptable) results. Generalization In recent years, we’ve found that applying SSL to highly general datasets improves the robustness and thus usefulness of our models at downstream tasks. As a result, the models the big labs are putting out are increasingly trained on self-prediction objectives over a diverse corpus of interleaved text, images, video and audio. By comparison, RL training has stayed quite “narrow”.

  5. Beating ARC the hard way

    ARC is benchmark developed to test out of distribution reasoning and common sense in general solvers. It is specifically designed to be: Easily solvable by most humans Not amenable to any kind of brute-force solvers (e.g. try every permutation of a solution) Not able to be solved with rote memorization The designers of ARC achieved the above in a creative way: by developing problems that contain visual puzzles in which the participant must find an algorithm that explains symmetries seen across several demonstrations. They then must apply that algorithm to a final input. This sounds complicated, but in practice it is quite intuitive – most children can complete ARC questions. LLMs are being pitched as general solvers, so lately we have been trying them out on this challenge. However, to make ARC amenable to being solved by a pure language model, you must remove the visual “clues” to the problem. More concretely, lets take the test problem on the ARC GitHub.

  6. General Intelligence (2024)

    Folks in the field of AI like to make predictions for AGI. I have thoughts, and I’ve always wanted to write them down. Let’s do that. Since this isn’t something I’ve touched on in the past, I’ll start by doing my best to define what I mean by “general intelligence”: a generally intelligent entity is one that achieves a special synthesis of three things: A way of interacting with and observing a complex environment. Typically this means embodiment: the ability to perceive and interact with the natural world. A robust world model covering the environment. This is the mechanism which allows an entity to perform quick inference with a reasonable accuracy.

  7. GPT-4o

    I’m very pleased to show the world GPT-4o. I came into the project mid-last year with Alexis Conneau with the goal of scaling up speech models and building an “AudioLM”. We knew we had something special late last year, but I don’t think either of us imagined that we’d able to pull off something as cool as GPT-4o in this short of a time frame. That came from the dedicated work of a core team of “believers”. I’m incredibly proud to have had the chance to work with so many talented and motivated people. I agree with Sam that interacting with this model feels like something new. I think what it boils down to is that for the first time ever, it “feels” better to interact with a computer program through speech rather than through text.

  8. Research Code

    At my job, I’m currently in a cycle that is involving working with software engineers quite a bit. One thing that has happened a number of times is that a software engineer will bring up “research code” with a condescending tone. The implication is that research code is messy, unreadable, and difficult to maintain. I don’t deny this! It often is those things, but I also think it has a beauty to its purpose and prose that is worth acknowledging. Most code has a purpose from the get go. Someone thinks “wouldn’t it be nice if my computer did <x>”, then designs and builds the code to achieve <x>.

  9. Learned Structures

    From 2019-2021, I was fascinated with neural network architectures. I think a lot of researchers in the field were at the time. The transformer paper had been out for a little while and it was starting to sink in how transformational it was going to be. The general question in the air was: what other simple tweaks can we make to greatly improve performance? As time has passed, I’ve internally converged on the understanding that there are only a few types of architectural tweaks that actually meaningfully impact performance across model scales. These tweaks seem to fall into one of two categories: modifications that improve numerical stability during training, and modifications that enhance the expressiveness of a model in learnable ways. Improving numerical stability is a bit of a black art.

  10. go/rulesofthumb

    Google has a neat internal website called “Rules of Thumb”, which compares the marginal cost of computational resources to the unit of a “SWE”. “SWE” refers to “Software Engineer” – which itself is the marginal cost to pay salary and benefits to the average engineer at the company. Throughout design docs at the company, you’ll see costs referred to in the units of SWE. For example, “deploying service <X> at 1000QPS will cost ~100 SWEs in resources”. I always thought comparing costs of fixed assets like compute, RAM, or database accesses to the cost of hiring a new employee was at once brilliant and dystopian. Brilliant because it allowed for some really interesting comparisons when thinking about how much effort to pour into optimizing some aspect of the stack – if you spent half a year optimizing something that saved 1 SWE of resources per year in a product that was growing, then that was a very good use of your time! It was also humbling to see how expensive each employee was – the unit of SWE came out to a rather ridiculous number of CPUs, RAM and/or hard disk space.

  11. Compute Multipliers

    I’ve listened to a couple of interviews with Dario Amodei, CEO of Anthropic, this year. In both of them, he dropped the term “compute multiplier” a few times. This concept is exceptionally important in the field of ML, and I don’t see it talked about enough. In this post, I’m going to attempt to explain what it is and why it is so important. Computational Efficiency Chinchilla is undoubtedly the landmark academic paper of 2022 in the field of Machine Learning. It’s most known for documenting the optimal relationship between the amount of compute poured into training a neural network and the amount of data used to train said network. In the process, it refuted some of the findings of an OpenAI paper from 2020, Scaling Laws for Neural Language models, which claimed that the optimal data:compute ratio was far smaller than was correct.

  12. Is the Reversal Curse a generalization problem?

    In my last post, I made a claim that the recently discovered reversal curse is not something that worries me. In fact, when I originally learned of it, I can’t say I was very surprised. In this post, I wanted to dig into that a little bit more. My hypothesis is that the reversal curse is a attribute of knowledge look-up, not a problem with the ability of LLMs to perform reasoning. Lookup in NNs Let me first describe how I think knowledge look-up in neural networks currently works. At a high level, autoregressive neural networks map inputs into high-dimensional vectors (we call them “latents”). At the terminus of the network, we map those latents into sets of probabilities which can be used to make a series of predictions through the process of sampling.

  13. The State of ML in 2023

    I’ve been trying to figure out how to best write this article for most of the last year. Today, I’ve decided to just write down something , rather than continue trying to wordsmith exactly what I mean. I am tremendously excited by everything that is going on in ML right now. The breadth of the problem space to which we can apply generalist learning techniques seems virtually unbounded, and every time we scale something up, we see new capabilities start to emerge. With all that being said, I think it’s worth considering from time to time what we haven’t achieved. Lets take a quick tour or the current state of the art: The image space I know quite well. With DALL-E 3, we cracked spelling (most of the time), which was a major capability gap of text-to-image models when compared to humans.

  14. DALL-E 3

    We released DALL-E 3 this week. It has been a labor of love for Aditya, Gabe and myself for a little over a year. It really is an impressive machine we have built. It continues to surprise me every day, despite having worked on it for so long. I’m extremely grateful to my fellow authors for a year of amazing learning and creating. I really hope everyone enjoys it and the world is a more colorful, graphical place because of it.

  15. ICML 2023

    I’ve met quite a few amazing people through this blog, most of which I’ve only had the chance to trade e-mails with. I’m attending ICML next week and would love to grab a coffee or beer with any of you. Shoot me an e-mail if interested. jbetker -at- gmail.

  16. On the efficiency of human intelligence

    A pet peeve of mine that often shows up in ML discourse is the claim that humans are much more data efficient at learning than the models we are currently training. The argument typically goes like this: “I’m blown away by how much knowledge my 3 year old has. They are smarter than most language models, despite being trained on a very small training dataset. Clearly, our models are missing something important because they cannot learn like my 3 year old!” But is the training dataset of a 3 year old actually smaller than a typical language model? For fun, I’d like to do some napkin math to bring the numbers down to levels that we can actually reason over. Starting with the LLM itself – let’s use Llama 65B.

  17. Techniques for debugging neural networks

    In my last post, I briefly discussed the infuriating fact that a neural network, even when deeply flawed, will often “work” in the sense that it’ll do above-random at classification or a generative network might create things that may sometimes look plausibly from the dataset. Given an idea that you’re testing out that is performing poorly – how, then, do you tell the difference between a botched implementation and an idea that just isn’t good? I think this is one of the toughest questions I have to deal with on a daily basis as an ML engineer. It’s the difference between funneling an immense amount of work into an idea that doesn’t pan out (which happens often!) or calling it early to look at something else. I definitely don’t have all the answers, but I have gathered a few tricks over the last couple of years that I wanted to share: Know how to interpret your loss curves. Different classes of NNs will have different loss curve shapes, but rarely do tweaks to a NN result in a change to the shape – generally performance will increase or decrease in a stepwise fashion.

  18. Ablations are really important

    I don’t read as many papers as I once did. I find this surprising as I always assumed that when I made ML my full-time job, I would spend a lot more time reading up on all of the things that other folks in the field are up to. To some extent, this is a weakness. There is a healthy balance one should strike between reading and writing and I’m definitely skewing a bit too far towards the writing side of things (code, not papers). With that said, I have the honor of working with some of the people I respect most in the field, and they don’t read much more than I do. There are several good reasons for this, but the one I’d like to talk about today is the importance of slow progress and ablations in the field. These two things work harmoniously to make papers either too boring to read past a quick glance or completely unusable for any future work.

  19. The “it” in AI models is the dataset.

    I’ve been at OpenAI for almost a year now. In that time, I’ve trained a lot of generative models. More than anyone really has any right to train. As I’ve spent these hours observing the effects of tweaking various model configurations and hyperparameters, one thing that has struck me is the similarities in between all the training runs. It’s becoming awfully clear to me that these models are truly approximating their datasets to an incredible degree. What that means is not only that they learn what it means to be a dog or a cat, but the interstitial frequencies between distributions that don’t matter, like what photos humans are likely to take or words humans commonly write down. What this manifests as is – trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point.

  20. GPT might be an information virus

    Obligatory: the views and opinions expressed in this post are my own and do not represent the views and opinions of my employer. In light of all the hype going around about ChatGPT, I wanted to offer my “hot take” on what the next 2-5 years of the web look like. One aspect of the rise of generative models that isn’t getting the right amount of attention is the long-term effects on the information economy. I think that being able to automatically produce arbitrary content that is indistinguishable from human-generated content at scale is the death knell of the web as we know it. The web today As we speak, thousands of small businesses are being started up with the sole purpose of exploiting ChatGPT as a “SEO expert”. It is being used to write websites, social media posts and reviews to promote products and businesses. For the last decade, this job has been tasked to content farms in poorer areas of the world.

  21. The Fundamental Building Blocks of DL

    I’m going to take a stab at nailing down what I believe to be the five fundamental components of a deep neural network. I think there’s value in understanding complex systems at a simple, piecewise level. If you’re new to the field, I hope that these understandings I’ve built up over the last few years help you! Data Representation The unit of data representation in a DNN is a vector. Vectors are called many different things: embeddings, tensors, activations, hidden states. They’re all just a list of floating point numbers that represent some single thing. Storage The learned weights of the neural network are where data is stored.

  22. Grokking Diffusion Models

    Since joining OpenAI, I’ve had the distinct pleasure of interacting with some of the smartest people on the planet on the subject of generative models. In these conversations, I am often struck by how many different ways there are to “understand” how diffusion works. I don’t think most folk’s understanding of this paradigm is “right” or “wrong”: they are just different. I think there is a distinct value in having a different viewpoints here: an engineers perspective might be more useful to deploy these things to real products, whereas a mathematicians conceptualization may aid improvements in the core technology. I’d like to jump through a few of these vantage points in this post, in the hope that I can help more people understand how this amazing technology works. Compute & Quality One aspect of generative modeling that has become quite clear to me is that finding principled ways to throw compute at a problem is a surefire way to improve output quality. This is round-about way of defining the concept of “scaling” in ML.

  23. I’ve Joined OpenAI

    I’ve been meaning to write this for a couple of months now, but simply haven’t found the time. Life has gotten quite busy for me lately, and I hope to explain why. First, the elephant in the room – I have left Google and finally stepped into the ML industry. I’ve accepted a position as a research engineer at OpenAI. To say that I am over the moon about this would be to understate it. This is, quite literally, my dream job. Somehow I have convinced someone to pay me to do the exact thing that I spend most of my free time doing anyways.

  24. The case for composite models

    In machine learning research, there is often a stated desire to build “end to end” training pipelines, where all of the models cohesively learn from a single training objective. In the past, it has been demonstrated that such models perform better than ones which are trained from multiple components, each with their own loss. The reasoning behind this notion is sound: every time you break up a model into different parts, you must necessarily introduce a new lossy medium: Every individual output is lossy. The prevailing theory is that these losses build up and produce an altogether inferior model at the end of the pipeline. I am not going to try to convince you that models trained separately from different losses (which I’ll call “composite models” in this post) are better than those trained end-to-end. Instead, I want to make the case for when they should be used. An few analogies A similar debate to this occurs across many engineering fields: In software, we debate the virtues of “monolithic architectures” versus “microservices”.

  25. Lab notes: Cheater latents

    Lab notes is a way for me to openly blog about the things I am building. I intend to talk about things I am building and the methods I plan to use to build them. Everything written here should be treated with a healthy amount of skepticism. I’ve been researching something this week that shows a lot of promise, and I really wanted to write about it. I call them “cheater latents”. They’re inspired by something I observed from Tortoise: in an early version of Tortoise, I trained the AR model by using the output clip itself as the conditioning input. What that looked like is this: Original Tortoise AR model training pipeline I was surprised by how well this worked in validation, but then quickly grew dismayed when I noticed that it had very poor generalization performance.

  26. Lab notes: Confidence decoders

    Lab notes is a way for me to openly blog about the things I am building. I intend to talk about things I am building and the methods I plan to use to build them. Everything written here should be treated with a healthy amount of skepticism. I wanted to write about something I built about a month ago that I think is really neat and I would like to return to someday. A quick disclaimer is that I think there is a strong probability that formal research on this idea already exists: I vaguely recall reading something similar about a year ago. If this is true and you would like to see credit given, drop me a line. I was thinking about an interesting problem recently: the wav2vec ASR model works using a sequence-to-sequence modeling scheme called “Connectionist Temporal Classification” .

  27. My deep learning rig

    A lot of people have asked about the computers I used to train TorToiSe. I’ve been meaning to snap some pictures, but it’s never “convenient” to turn these servers off so I keep procrastinating. We had some severe thunderstorms today here in the front range which forced me to shut down my servers. I took the opportunity to take some photos. A little history to start Building out my servers has been a long, multi-year effort. I started the process right around the time that NVIDIA launched their Ampere GPU lineup. I was fortunate to be able to grab 6 RTX 3090s right around launch time, and slowly built up my inventory one GPU at a time over the course of 2021.

  28. Friends don’t let friends train small diffusion models

    For my next project, I want to play around in the music generation space. I think it’ll be interesting to apply some of the lessons learned building Tortoise to music. The first step is building the musical equivalent of a vocoder: a model that will transform a MEL spectrogram to waveform data. That way the main generator(s) can work in highly reduced spectrogram space, just like Tortoise. I could just train a new Univnet model. That probably would have been the wisest choice. However, I don’t really like training GANs and I have no experience training Univnet.

  29. TorToiSe Architectural Design Doc

    Overview TorToiSe is a text-to-speech (TTS) program which can mimic voices given 2-4 examples. It is composed of five separately-trained neural networks that are pipelined together to produce the final output. This document will first go into details about each of the five models that make up Tortoise, and will wrap up with a system-level description of how they interoperate. The Autoregressive Decoder Reference Clips A list of reference clips are also provided to the model. The model uses these clips to figure out how to properly mimic the voice, intonation, prosody, etc of the speech it is expected to produce. In code and the diagrams, the reference clips are referred to as “conditioning” inputs. The reference clips are processed by a small model architecture reminiscent of ViT : A stack of reducing convolutions followed by a stack of full-attention transformers.

  30. Surrogate Losses for Diffusion Models

    As I covered in my last post , I’m currently working on improving the quality of the diffusion model used to rebuild discretized audio signals for tortoise-tts. Since realizing that the diffusion model can work entirely with spectrograms, I have been re-structuring the model to be a flat transformer/resnet hybrid . One nifty thing about this set-up is that I can now concatenate the diffusion inputs with the conditioning signal from the discretized audio signal and feed the whole thing into the model. This is the same thing that multiple authors working on diffusion models have done with the low-resolution inputs for super-resolution models. I naively figured that I could simply plop an embedding of the discretized audio signal into the model inputs. I found out that this was wrong. The diffusion model would ignore the conditioning signal and would learn to produce gibberish that sounded like human speech but had no actual words.

  31. Improving Diffusion Models for TTS

    I’ve spent the majority of the last two months working on improving the diffusion model in Tortoise TTS . The model used in v1 had a few major shortcomings: Conditioning inputs were bottlenecked to a very small dimensional input into the main model, limiting their effectiveness. The model was trained on audio signals at 11kHz. To make this feasible, I needed to chop up the signals into small clips which have limited context, and: The model itself was relatively shallow: the top layers only had a depth of 1 and model channels at the top levels were very restricting. No processing was performed on the input codes, meaning the main u-net needed to do all the heavy lifting here. I tried out many solutions to the above problems. Of them, the spatial dimensionality of the model was the hardest to tackle.

  32. Tortoise TTS Update

    I’ve updated the tortoise-tts repo with a script that automatically download model weights (thank to the HuggingFace Hub for hosting them!). I’ve also created a colab notebook if you want to try this out on Google hardware. Make sure you pick a GPU runtime. Sample outputs can be found in the results/ folder of the GitHub repo. Find some handpicked generate below. I’m not done with this project. It is clear to me that the autoregressive model does an extremely good job at producing realistic prosody.

  33. DALL E for TTS: TortoiseTTS

    In an earlier post , I walked you through a project I’ve been working on, which I called “triforce” at the time. I’ve finished training a first pass on this collection of models and want to write about the results. Deploying this speech CLIP model on the outputs of my autoregressive speech token generator made all of the difference. Outputs are consistently awesome, and almost always clearly convey the desired speech. Adding CLIP to the ensemble After training the three triforce models, I was having considerable difficulty with the autoregressive portion of the model. Specifically, while I would generate a lot of really good speech, I would also regularly generate audio with a single syllable that dragged, for example: “three pigs went to the paaaaaaaaaaaaaaaaaaaaaaaa”. After spending some time tinkering with different methods of autoregressive generation (thanks for all your work, HF team!), I finally came around to the realization that the secret sauce is adding a fourth model to the ensemble that serves the same purpose as DALLE’s CLIP.

  34. Batch speech transcription with ocotillo

    As I mentioned in my previous blog post, I’m currently working on text-to-speech models. I’m taking the “scale-it-to-the-moon” approach, so I need a lot of data. Fortunately, speech data is pretty easy to come by. Audio books, podcasts, YouTube and large archives of speeches and presentations are available all over the internet. The problem is that this audio generally isn’t transcribed. As you may know from interacting with Alexa, her spin-offs, or your smartphone assistant, digital speech recognition software is pretty damned good these days. My goal was to leverage this to build a huge dataset of artificially transcribed audio clips.

  35. Triforce: A general recipe for kickass Generative Models

    For the past two years, I’ve been tinkering around with generative models in my spare time. I think I’ve landed on an approach that produces by far the most compelling results available today, and which scales like big language models. I’d like to outline the approach here. First of all, I want to touch on something that’ll become immediately obvious: this isn’t a novel architecture or anything. In fact, it is pretty much OpenAI’s DALL E with a diffusion upsampler attached. Instead, it’s a way of thinking how one can (1) improve upon DALL E and (2) universally model generative domains using a single set of techniques. Three Models This approach uses three different neural networks to produce the finished result, all trained separately from one another.

  36. Switched Convolutions – Spatial MoE for Convolutions

    Switched Convolutions – Spatial MoE for Convolutions Abstract I present switched convolutions: a method for scaling the parameter count of convolutions by learning a mapping across the spatial dimension that selects the convolutional kernel to be used at each location. I show how this method can be implemented in a way that has only a small increase in computational complexity. I finally discuss applications of switched convolutions and show that applying them to a pre trained VAE results in large gains in performance. I have open sourced all of my work on switched convolutions. It can be found here . Background Despite the growing popularity of autoregressive models based on Transformers for image processing tasks, CNNs remain the most efficient way to perform image processing. One disadvantage of CNNs is that it is difficult to effectively scale their parameter count.

  37. SRGANs and Batch Size

    Batch size is one of the oldest hyper parameters in SGD, but it doesn’t get enough attention for super-resolution GANs. The problem starts with the fact that most SR algorithms are notorious GPU memory hogs. This is because they generally operate on high-dimensional images at high convolutional filter counts. To put this in context, the final intermediate tensor of the classic RRDB model has a shape of (<bs>x64x128x128) or over 33M floats at a batch size of 32. This one tensor consumes more than 10% of the models total memory usage! To cope with this high memory usage, SR papers often recommend training with miniscule batch sizes in the regime of 4-16 samples per batch. This is wholly inadequate, as I will discuss in this article.

  38. Training SRFlow in DLAS (and why you shouldn’t)

    SRFlow is a really neat adaptation of normalizing flows for the purpose of image super-resolution. It is particularly compelling because it potentially trains SR networks with only a single negative-log-likelihood loss. Thanks to a reference implementation from the authors or the paper, I was able to bring a trainable SRFlow network into DLAS . I’ve had some fun playing around with the models I have trained with this architecture, but I’ve also had some problems that I want to document here. First of all – the good First of all – SRFlow does work. It produces images that are perceptually better than PSNR-trained models and don’t have artifacts like GAN-trained ones. For this reason, I think this is a very promising research direction, especially if we can figure out more effective image processing operations that have tractable determinants.

  39. Translational Regularization for Image Super Resolution

    Abstract Modern image super-resolution techniques generally use multiple losses when training. Many techniques use a GAN loss to aid in producing high-frequency details. This GAN loss comes at a cost of producing high-frequency artifacts and distortions on the source image. In this post, I propose a simple regularization method for reducing those artifacts in any SRGAN model. Background on SR Losses Most SR models use composite losses to achieve realistic outputs. A pixel-wise loss and/or a perceptual loss coerces the generator to produce images that look structurally similar to the input low-resolution image. With only these losses, the network converges on producing high-resolution images that are essentially the numerical mean of all of the training data.

  40. Deep Learning Art School (DLAS)

    At the beginning of this year, I started working on image super-resolution on a whim: could I update some old analog-TV quality videos I have archived away to look more like modern videos? This has turned out to be a rabbit hole far deeper than I could have imagined. It started out by learning about modern image super-resolution techniques. To this end, I started with a popular GitHub repo called ‘mmsr’. This repo no longer exists, and has since been absorbed into mmediting , but at the time it was a very well-written ML trainer library containing all of the components needed to set up an SR-training pipeline. As my SR (and GAN) journey continued, I often needed to make sweeping alterations to the trainer code. This frustrated me, because it invalidated old experiments or added a ton of labor (and messy code) to keep them relevant.

  41. Accelerated Differentiable Image Warping in Pytorch

    Computing optical flow is an important part of video understanding. There are many ways to train a model to compute this, but one of the more compelling methods is to: Feed a model an image pair Have it predict optical flow Apply that optical flow to the original image Compute a pixel-wise loss against the second image. In order to use this algorithm, however, you need a differentiable way to do step (3), typically called an “image warp”. Tensorflow has just such an operation in contrib, but to my knowledge Pytorch does not. After digging around for awhile today, I found what I needed in one of nVidia’s open source repositories: https://github.com/NVIDIA/flownet2-pytorch In this repository, the author has implemented a new CUDA primitive called “ resample2d “. Although there isn’t any documentation on this operation, it is exactly what is needed to compute an image warp given an optical flow vector. Suppose you have an image and a .flo file, which you can find from several places.

  42. Batch Normalization is a Hack

    Batch normalization has a simple goal: stabilize the gradients of large computational graphs. In doing so, this technique has enabled the deep learning renaissance that almost every major ML breakthrough in the last 5 years has relied on. The concept is sound: by regularizing the mean and variance of the inputs of nearly every layer in a neural network, the gradients of that network rarely explode backward pass. The end result is that many neural networks can be easily trained with gradient techniques that would otherwise have never converged. So why am I calling it a hack? Let’s dig in. It violates batch invariance When training a neural network with SGD, we often use mini-batches.

  43. Diving into Super Resolution

    After finishing my last project , I wanted to understand generative networks a bit better. In particular, GANs interest me because there doesn’t seem to be much research on them going on in the language modeling space. To build up my GAN chops, I decided to try to figure out image repair and super-resolution. My reasoning was actually pretty simple: I have a large collection of old VHS quality Good Eats episodes that I enjoy watching with my family. Modern flat screens really bring out how inadequate the visual quality of these types of old videos are, however. Wouldn’t it be great if I could use machine learning to “fix” these videos to provide a better experience for myself and my family? How hard could it be?

  44. Fine-tuning XLNet For Generation Tasks

    About a month ago, I decided to take the plunge into learning how to fine tune a language generation model. One use-case of language generation that I found particularly compelling was abstractive document summarization. A lot of the papers currently available that deal with abstractive summarization and transformers work by truncating the input text to the maximum sequence length of the model. In the post-transformer XL world, I thought it’d be neat to fix that limitation. XLNet and TransformerXL are the two recurrent language models currently available in the Transformers NLP library . “Recurrent” in this context means that they were designed to model extremely long sequences by breaking those sequences into chunks and processing them one at a time. The chunks are then tied together via a “memory” that is recursively passed between from forward pass to the next.

  45. Learning to Learn: My Second Foray Into Machine Learning

    My desire to understand how the mind works started when I was choosing what I wanted to do in college, in 2000. Back then I was a nerdy kid who was pretty good with computers, but who had grown an insatiable interest for figuring out how the mind ticked. Not knowing a whole lot about the world, I figured my way into progressing this puzzle was the field of psychology. As a result, I joined UCSB as a biology major, with an expressed interest in both psychology as well as psychiatry. Two years later, my passion for working with computers overtook my desire to enter into the far murkier fields of the mind. Computer science was easy for me. Something about the problems spoke to me: the solutions just appeared in my mind without too much thought.