Booting portfolio system
20%
← Return to work

project

2D Autonomous Driving

A reinforcement-learning comparison of PPO, A2C, and QRDQN agents trained to navigate simulated highway and merge environments.

01
research
02
Researcher and Developer
03
Python / PyTorch / Stable-Baselines3 / Highway-Env / TensorBoard / BoTorch

This project compares PPO, A2C, and QRDQN reinforcement-learning agents in highway-env, using simulated highway and merge driving tasks to test both baseline behavior and the effect of hyperparameter optimization.

The goal was to see how the agents performed when trained and evaluated in the same driving environment, and how well that performance transferred when the test environment introduced merging traffic.

A2C agent navigating traffic in the highway-env merge scenario.
As the model nears the car in front, it manages to slow down to avoid crashing. It even slows down when the merge happens, even though the merge is far away, to guarantee no crash.

Setup

I tested three configurations:

  • Configuration A: train on default-fast, test on default.
  • Configuration B: train on default-fast, test on merge.
  • Configuration C: train on merge, test on merge.

Each algorithm used the Stable-Baselines3 implementation with the default CnnPolicy and grayscale observations from highway-env. Training runs used 100,000 learning steps and 20 evaluation trials. Hyperparameter optimization used Bayesian optimization over learning rate, discount factor, and RMSProp for A2C, with each tuning trial trained for 50,000 timesteps and evaluated over 10 trials.

Results

Before tuning, all models handled the straight highway task better than the merge transfer task. A2C was the strongest baseline in configuration A and also held up best when tested on merge after training on the default environment.

After hyperparameter optimization, configuration A improved substantially:

| Algorithm | Before HPO | After HPO | |---|---:|---:| | A2C | 42.9 | 56.64 | | PPO | 30.92 | 48.46 | | QRDQN | 31.94 | 56.46 |

QRDQN improved the most in configuration A, reaching nearly the same final reward as A2C after tuning. PPO also improved strongly. A2C improved less by percentage, but started from the strongest baseline.

Configuration B moved in the opposite direction:

| Algorithm | Before HPO | After HPO | |---|---:|---:| | A2C | 29.51 | 29.01 | | PPO | 21.54 | 13.21 | | QRDQN | 19.55 | 11.66 |

For the merge transfer task, tuning did not improve the tested agents. A2C remained roughly stable, while PPO and QRDQN both declined after optimization.

Takeaway

The main result is that hyperparameter optimization helped in the environment it was tuned around, but did not transfer cleanly to the more dynamic merge test. Configuration A showed large gains after tuning, especially for QRDQN. In configuration B, the tuned policies were less robust, which suggests the tuning process was strongly environment-dependent.

The next useful step would be a fuller optimization pass for configuration C, where the agents are trained and tested directly in the merge environment.