Eric Jang

Here are some of my projects and publications. You can find more coding projects on my Github profile.


7/2020 Meta-Learning Requires Meta-Augmentation

Data augmentation techniques are ubiquitous in ML and often result in a substantial improvement in performance. We formalize "meta-augmentation" and show that you can apply it to pretty much any meta-learning problem and any meta-learner.

6/2020 RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer

When applying Sim2Real to robotic tasks, it's crucial to have the same objects in one domain consistently show up in the other domain. We enforce this by ensuring consistency of a pretrained object detector's activations in the sim and real domains.

3/2020 Scalable Multi-Task Imitation Learning with Autonomous Improvement

This was follow-up work to Watch-Try-Learn. We improve a multi-task policy by self-imitating re-labeled policy rollouts. Re-labeling is done via a learned task embedding function.

2/2020 Thinking While Moving

Most RL algorithms assume that the world pauses and waits for agents to compute a decision. We remove this limitation for value-based RL methods, achieving a 49% speedup on the QT-Opt grasping system.

5/2019 Watch, Try, Learn: Meta-Learning from Demonstrations and Rewards

We train an agent to imitate visual manipulation skills from raw pixels. The agent meta-learns how to re-try from failures, allowing it to fix its own mistakes in a single trial. Twitter Thread

10/2018 Grasp2Vec: Learning Object Representations from Self-Supervised Grasping

Robots that teach themselves instance grasping without any labeling. Twitter Thread

10/2018 Generative Ensembles for Robust Anomaly Detection

We use generative models to perform out-of-distribution detection. Concurrently with Nalisnick '18, we show that likelihood models can suffer OoD problems when performing OoD tasks and improve their robustness with uncertainty estimation via ensembling.

6/2018 QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

We use reinforcement learning to learn grasping from RGB images on real robots. Robot learns emergent manipulation behaviors like retrial and object singulation!.

12/2017 Sim2Real View Invariant Visual Servoing by Recurrent Control

Careful camera & workspace calibration is not always possible when deploying robots in the real world. In this work we use a recurrent policy to adapt to unknown optical distortions in a robotic reaching task.

9/2017 Deep Reinforcement Learning for Vision-Based Robotic Grasping

We came up with a simulated benchmark for robotic bin picking from RGB images and evaluated 6 off-policy learning algorithms on it.

9/2017 Time Contrastive Networks

We came up with a really data-efficient, unsupervised feature representation for robotic imitation learning. Using this as a state representation, we can get a robot to imitate a human pouring liquids with a single video demonstration (!!).

7/2017 End-to-End Learning of Semantic Grasping

We taught a bunch of robotic arms to learn how to 1) detect 2) classify 3) pick up a variety of objects in a nearly autonomous setup.

11/2016 Categorical Reparameterization with Gumbel-Softmax

A method for backpropagation through discrete categorical samples in neural networks.

11/2016 Emergence of selectivity to looming stimuli in a spiking network model of the optic tectum

A biophysical spiking neural network model of collision detection in Xenopus tadpoles.


Path Tracer (JAX)

A differentiable path tracer written in JAX.

Normalizing Flows (JAX)

A normalizing flows implementation written in JAX.


A MAML implementation written in JAX.

ShaderToy Profile

Realtime graphics artwork on ShaderToy.


Visual Debugger for Deep Learning, built on TensorFlow.

Recurrent Neural Networks : a Dynamical Systems Perspective

Final project for my Chaotic Dynamics class (APMA136). I extended a paper to analyze the geometric structure of the submanifold dynamics of Echo State Networks.

Simulating self-organization of color and orientation maps in the visual cortex

Final project for my Neural Dynamics class (APMA2821V). I extended an existing biophysical cortex model to self-organize functional orientation maps and ocular dominance, alongside color orientation maps.

High Frequency Bitcoin Arbitrage

A high frequency trading bot that exploits price inefficiencies between bitcoin/altcoin markets for riskless profit. Does pairwise and triangular arbitrage.

GPU Snow

Three team members and I implemented Disney's Material Point Simulation paper from scratch and extended it to work on the GPU.

3x3 SVD

CPU and CUDA implementations of SVD of 3x3 matrices in 2 microseconds.

Book Crooks

Fall 2013 PennApps submission. We found a JavaScript exploit in Google Books that allowed us to download the full version of any book scanned by Google. Code not released for obvious reasons.

Painting Procedural Render-time Grass

Final project for the Pixar Undergraduate Program. I wrote a Renderman DSO in C++, RIB, Python, and RSL that allows a user to direct vegetation from the rendered perspective. Grass driven by Bezier curves and fractal brownian motion.

Synfire Braid Dynamics in Spatially Constrained Networks

A classmate and I looked at how the physical spatial arrangement of neurons could change the polychronization dynamics of neural networks.

Embodied Simulation of Reinforcement Learning and Motor Control

I wrote a spiking neural network simulator from scratch in Python and used it to model the dopaminergic-thalamic circuit. I coupled the neural simulation to a simulated robot created in Blender.

The Social Docking Project

Volunteer browser-powered distributed computing running on a $0 budget. Gasteiger Partial Charge Equalization, General AMBER Force Field, and Monte Carlo conformation search implemented in JavaScript.