RL Car Racing: DQN Agent
Implemented a Deep Q-Network agent that learns to drive autonomously in OpenAI's CarRacing-v2 environment, combining deep learning with reinforcement learning for continuous visual control.
- ▸Deep Q-Network learning autonomous driving from raw pixel observations in a continuous control environment
- ▸Systematic comparison of batch sizes and learning rates using saved checkpoints, reward plots, and agent visualizations
- ▸Image preprocessing pipeline crops each observation to 84×84 grayscale and stacks four frames for temporal context
Overview
Final project for the Reinforcement Learning Practical course at the University of Groningen, supervised by Prof. Matthia Sabatelli. Built with my partner Konstantinos Chasiotis.
The challenge: teach an agent to drive a car around a procedurally generated track using only pixel observations, no access to car physics, no hand-crafted features. The agent must learn to steer, accelerate, and brake purely from visual input through trial and error.
Approach
We implemented a Deep Q-Network (DQN) for the CarRacing-v2 environment. The key challenge is that the observation space is high-dimensional (96x96 RGB pixels) and the environment requires precise continuous control, making it significantly harder than classic RL benchmarks like CartPole or Atari games with discrete, low-dimensional states.
The DQN discretizes the action space and uses a convolutional neural network to map raw pixel frames to Q-values for each possible action. The agent learns which actions maximize cumulative reward (staying on track, maintaining speed) through experience replay and temporal difference learning.
Technical Details
- State representation: Raw pixel observations from the environment (96x96 RGB), preprocessed and stacked for temporal context
- Network architecture: CNN processing visual input → fully connected layers mapping to discrete action Q-values
- Training: Experience replay buffer for sample decorrelation, target network for training stability, epsilon-greedy exploration
- Hyperparameter tuning: Systematic comparison of batch sizes and learning rates, with saved checkpoints, reward plots, and agent visualizations documented in analysis notebooks
Results
The repository includes trained checkpoints and visualizations showing how different batch sizes and learning rates affect the agent's driving behavior:
What I Learned
- Visual RL is a fundamentally different challenge from state-based RL: the representation learning problem dominates
- Experience replay and target networks are central to stabilizing value estimates during training
- Batch size and learning rate materially change the learning curve and observed driving behavior
- Preprocessing choices such as grayscale conversion, cropping, frame skipping, and frame stacking make visual RL substantially more tractable