the last question
RL is dumb but works
Apr 19, 2026
Comma’s controls challenge asks for a steering controller that follows a target lateral acceleration.
The README gives each rollout two costs.
\[\small \mathtt{lataccel\_cost} = \frac{\sum(\mathrm{actual}-\mathrm{target})^2}{\mathrm{steps}}\times 100\] \[\small \mathtt{jerk\_cost} = \frac{\sum\left(\frac{\mathrm{actual}_t-\mathrm{actual}_{t-1}}{\Delta t}\right)^2}{\mathrm{steps}-1}\times 100\]Here actual and target refer to lateral acceleration. \(\Delta t\) gives the time between steps. The first cost penalizes tracking error. The second penalizes rapid changes in acceleration. The controller must minimize their weighted total.
Comma supplies the driving data, simulator and a PID baseline. I trained a neural network controller using imitation learning followed by reinforcement learning.
Inputs and output
Lateral acceleration is the sideways acceleration of a turning car. The controller receives the target and current lateral acceleration, speed, forward acceleration, road bank and nearly five seconds of upcoming targets and road conditions.
The output is a steering command between -2 and 2, not an angle in degrees. The simulator applies it and returns the next lateral acceleration. The controller does not control speed.
Comma includes this PID controller.
from . import BaseController
class Controller(BaseController):
def __init__(self):
self.p = 0.195
self.i = 0.100
self.d = -0.053
self.error_integral = 0
self.prev_error = 0
def update(self, target_lataccel, current_lataccel, state, future_plan):
error = target_lataccel - current_lataccel
self.error_integral += error
error_diff = error - self.prev_error
self.prev_error = error
return self.p * error + self.i * self.error_integral + self.d * error_diff
controller = Controller()
command = controller.update(0.2, 0.1, state=None, future_plan=None)
# First call returns about 0.0242.
The simulator calls update once per step. target_lataccel and current_lataccel are numbers. state holds road bank, speed and forward acceleration. future_plan holds the upcoming targets and road conditions. This PID uses only the two acceleration arguments. It returns one steering number, which the simulator clips to [-2, 2].
Comma’s pretrained network simulates the car’s response. My network chooses the steering command. The simulator stays fixed during controller training.
The required steering depends on speed, road bank and recent motion, so the input includes the current state and its history.
The PyTorch excerpts omit file loading, GPU buffer management and logging. B is the number of parallel simulations. [B, 256] means one row per car with 256 numbers per row. For one car, B = 1.
Early controllers
I first tried model predictive control. The idea was to simulate several possible steering sequences, score them and use the first command from the best one. An early ChatGPT-generated version used this prediction.
next_lataccel = current_lataccel + steering * dt
This model ignored speed, road bank and history. The controller oscillated even after changes to the optimizer’s smoothing weight.
I implemented proximal policy optimization, or PPO, on LunarLander before applying it to steering. A policy maps inputs to actions. PPO updates it using rewards from simulated runs.
The first steering network learned to imitate PID. PID combines tracking error, accumulated error and the error’s rate of change. Training a network to reproduce its commands is behavioral cloning.
I added curvature to the inputs by subtracting the road-bank contribution from lateral acceleration and dividing by speed squared. This describes how tightly the road bends. I retained the raw speed and acceleration as well.
Observation and network
The final controller receives 256 numbers. Sixteen describe the current state and tracking error. Forty hold the previous 20 steering commands and 20 lateral accelerations. The remaining 200 hold four channels of road preview, each padded to 50 values. The preview contains target acceleration, road bank, speed and forward acceleration.
The inputs below are float32 tensors. state contains five measurements. preview has 50 future timesteps and four channels. Histories run from oldest to newest, with error_history including the latest error. bank is acceleration caused by road bank, not the bank angle.
import torch
from torch import nn
from torch.nn import functional as F
from torch.distributions import Beta
def make_observation(state, steer_history, lat_history,
error_history, preview):
# state [B, 5], histories [B, 20], preview [B, 50, 4]
target, current, speed, accel, bank = state.unbind(dim=1)
previous_steer = steer_history[:, -1] # [B]
speed_squared = speed.square().clamp_min(1.0)
target_curve = (target - bank) / speed_squared
current_curve = (current - bank) / speed_squared
combined_accel = torch.sqrt(current**2 + accel**2) / 7.0
core = torch.stack([
target / 5,
current / 5,
(target - current) / 5,
target_curve / 0.02,
current_curve / 0.02,
(target_curve - current_curve) / 0.02,
speed / 40,
accel / 4,
bank / 2,
previous_steer / 2,
error_history.mean(dim=1) * 0.1 / 5,
(preview[:, 0, 0] - target) / 0.1 / 5,
(current - lat_history[:, -1]) / 0.1 / 5,
(previous_steer - steer_history[:, -2]) / 0.1 / 2,
combined_accel,
(1 - combined_accel).clamp_min(0),
], dim=1) # [B, 16]
obs = torch.cat([
core, # [B, 16]
steer_history / 2, # [B, 20]
lat_history / 5, # [B, 20]
preview[:, :, 0] / 5, # [B, 50] target acceleration
preview[:, :, 1] / 2, # [B, 50] road bank contribution
preview[:, :, 2] / 40, # [B, 50] speed
preview[:, :, 3] / 4, # [B, 50] forward acceleration
], dim=1)
return obs.clamp(-5, 5) # [B, 256]
stack makes the 16 features into columns. cat joins them to the history and preview. The divisions put quantities such as speed and curvature on comparable scales.
I feed it into four hidden layers of 256 units each. Each linear layer multiplies its input by learned weights and adds a learned bias. ReLU sets negative results to zero. Without ReLU, the whole stack could be replaced by one linear layer.
def make_network(outputs, final_gain):
layers = []
for _ in range(4):
layers += [nn.Linear(256, 256), nn.ReLU()]
layers += [nn.Linear(256, outputs)]
network = nn.Sequential(*layers)
for layer in network:
if isinstance(layer, nn.Linear):
nn.init.orthogonal_(layer.weight, gain=2**0.5)
nn.init.zeros_(layer.bias)
nn.init.orthogonal_(network[-1].weight, gain=final_gain)
return network
actor = make_network(outputs=2, final_gain=0.01)
critic = make_network(outputs=1, final_gain=1.0)
# actor [B, 256] -> four hidden layers -> [B, 2]
# critic [B, 256] -> four hidden layers -> [B, 1]
The actor chooses steering and has 263,682 parameters, initialized with random weights and zero biases. The critic estimates future reward. It receives the same inputs but has separate weights.
The actor chooses a steering change limited to 0.25 per step. Adding it to the previous command restricts exploration to nearby commands.
Its two outputs define a Beta distribution. Sampling from it lets the controller test different steering changes during training.
def steering_distribution(obs):
out = actor(obs) # [B, 2]
alpha = F.softplus(out[:, 0]) + 1 # [B]
beta = F.softplus(out[:, 1]) + 1 # [B]
return Beta(alpha, beta)
obs = make_observation(
state, steer_history, lat_history, error_history, preview
) # [B, 256]
previous_steer = steer_history[:, -1] # [B]
with torch.no_grad():
dist = steering_distribution(obs)
x = dist.sample() # [B], between 0 and 1
delta = 0.25 * (2 * x - 1) # [B], between -0.25 and 0.25
steering = (previous_steer + delta).clamp(-2, 2)
old_logp = dist.log_prob(x) # [B]
value = critic(obs).squeeze(-1) # [B]
softplus converts each raw output into a positive number. Adding 1 keeps both Beta parameters above 1. The resulting distribution chooses a number between 0 and 1. If it samples x = 0.6, the steering change is 0.25 * (2 * 0.6 - 1) = 0.05. With a previous command of 0.10, the new command is 0.15.
The simulator receives steering and returns the next current acceleration. The chosen command and previous acceleration reading enter the histories for the next observation. Each step represents 0.1 seconds.
old_logp is the sampled action’s log probability density, saved for the PPO update. no_grad collects experience without building a backpropagation graph.
Imitation learning
The final pretraining stage uses recorded steering from the supplied CSVs instead of PID. Examples come from the start of each route, before learned control begins. This trains the randomly initialized actor before PPO.
Reversing the action conversion maps a recorded change of 0.05 to a Beta target of 0.6. M is the number of examples in a training minibatch. The commands already have their sign converted from the CSV convention.
bc_optimizer = torch.optim.AdamW(
actor.parameters(), lr=0.01, weight_decay=1e-4
)
# bc_obs [M, 256], logged_steer [M], logged_previous_steer [M]
raw_target = ((logged_steer - logged_previous_steer) / 0.25)
raw_target = raw_target.clamp(-1, 1)
x_target = ((raw_target + 1) / 2).clamp(1e-6, 1 - 1e-6)
dist = steering_distribution(bc_obs)
bc_loss = -dist.log_prob(x_target).mean() # scalar
bc_optimizer.zero_grad()
bc_loss.backward()
nn.utils.clip_grad_norm_(actor.parameters(), 2.0)
bc_optimizer.step()
The loss decreases when the actor assigns more probability density to recorded changes. backward calculates how each weight affects the loss. The optimizer uses these gradients to adjust the weights. This update trains on examples without running the simulator.
PPO training
For PPO, the actor controls the simulator. Each step records the inputs, sampled action, log probability density and critic value. The reward is the negative cost of tracking error and jerk.
The scored window is 40 seconds after warmup. With a 0.1-second timestep, T = 400. Observations have shape [B, T, 256]. Actions, saved log probabilities and values each have shape [B, T].
The default reward uses the acceleration produced after each command.
# predicted and target are post-step arrays [B, T], T = 400
tracking_cost = 100 * 50 * (target - predicted).square()
jerk = torch.diff(
predicted, dim=1, prepend=predicted[:, :1]
) / 0.1
rewards = -(tracking_cost + 100 * jerk.square()) # [B, T]
The factor of 100 scales both costs. Tracking error has an additional weight of 50. Prepending the first prediction makes the first jerk term zero, excluding the warmup boundary.
The reference score averages 399 jerk differences. This 400-step reward vector includes the initial zero.
Steering affects later steps as well as the current one. The critic estimates the remaining discounted reward, where rewards further in the future count less.
The backward calculation below compares rewards and later value estimates with the critic’s prediction. The result is an advantage, an estimate of how much better or worse the outcome was than expected.
def training_targets(rewards, values, reward_std):
# Both inputs are [B, T], collected without gradients.
rewards = rewards / max(reward_std, 1e-8)
B, T = rewards.shape
advantage = torch.empty_like(rewards)
running = torch.zeros_like(rewards[:, 0]) # [B]
for t in reversed(range(T)):
next_value = values[:, t + 1] if t + 1 < T else 0
error = rewards[:, t] + 0.95 * next_value - values[:, t]
running = error + 0.95 * 0.9 * running
advantage[:, t] = running
returns = advantage + values # [B, T], targets for the critic
return advantage, returns
reward_std rescales rewards by their running standard deviation. The loop combines prediction errors across steps. 0.95 is the discount and 0.9 controls how much later errors contribute. This is generalized advantage estimation. The final step has no next value because the scored window ends there.
PPO uses the advantages to change the actor. I flatten the car and time axes together, so [B, T, 256] becomes [B * T, 256]. I flatten the saved actions and training targets in the same order, then shuffle matching rows into minibatches of M decisions.
pi_optimizer = torch.optim.Adam(actor.parameters(), lr=3e-4, eps=1e-5)
vf_optimizer = torch.optim.Adam(critic.parameters(), lr=3e-4, eps=1e-5)
def ppo_minibatch(obs_mb, x_mb, old_logp_mb, adv_mb, returns_mb):
# obs_mb [M, 256]. Every other argument is [M].
adv_mb = (adv_mb - adv_mb.mean()) / (adv_mb.std() + 1e-8)
dist = steering_distribution(obs_mb)
new_logp = dist.log_prob(x_mb)
ratio = (new_logp - old_logp_mb).exp() # [M]
actor_loss = -torch.minimum(
ratio * adv_mb,
ratio.clamp(0.8, 1.2) * adv_mb,
).mean()
predicted_value = critic(obs_mb).squeeze(-1) # [M]
critic_loss = F.mse_loss(predicted_value, returns_mb)
pi_optimizer.zero_grad()
vf_optimizer.zero_grad()
(actor_loss + critic_loss).backward()
nn.utils.clip_grad_norm_(actor.parameters(), 0.5)
nn.utils.clip_grad_norm_(critic.parameters(), 1.0)
pi_optimizer.step()
vf_optimizer.step()
The ratio compares the action’s probability density under the updated actor with its density during collection. 1.1 means a 10 percent increase. Positive advantages favor an increase, negative advantages a decrease. Clipping limits the incentive for large changes without imposing a hard bound on the weights.
The critic minimizes squared error against returns_mb. The full trainer also penalizes overly narrow action distributions. Saved observations, actions, log probabilities and targets remain fixed during the update.
Gradients are computed through the actor and critic, not the simulator.
Each batch gets four optimization passes before collecting new runs with the updated actor.
Training failures
Early PPO runs made the pretrained controller worse.
The imitation loss initially trained only the mean action. The mean could match the teacher while the distribution remained wide, producing poor commands when PPO sampled from it.
Training on the recorded action’s likelihood adjusted both the mean and spread.
The reward also omitted the tracking multiplier and the timestep divisor in jerk. This changed the tradeoff between tracking and smoothness from the challenge’s scoring rule.
I also scored actions against the previous acceleration. At step 100, the controller reads acceleration from step 99, but its command produces acceleration for step 100.
The corrected reward uses post-step acceleration paired with the target at that step. Jerk is calculated only within the scored window.
Scores improved across these fixes and other input and training changes. I did not isolate each change in a separate comparison.
Batching and search
To reduce simulation time, I batched routes, distributed work across MacBooks and moved the simulator and controller loop onto a GPU. The implementation is covered in Physics Go Brrr.
I also tested search around the policy. It proposed 16 candidates and simulated five steps ahead. For one checkpoint, this improved the score from 45.552 to 44.834, but evaluation time grew from about two minutes to 56 minutes.
After reward and training changes, the policy scored 43.729 without search, using one network call per step.
Repeated routes
To compare attempts under the same conditions, I used 500 routes with ten attempts each. Repeats shared the simulator’s random draws but sampled different policy actions. At each route and timestep, I subtracted the mean advantage across the ten attempts before updating the actor.
# B = 500 routes * 10 attempts. Each attempt has T = 400 steps.
advantage, returns = training_targets(rewards, values, reward_std)
grouped = advantage.reshape(500, 10, 400)
advantage = grouped - grouped.mean(dim=1, keepdim=True)
# Keep every array in the same route, attempt, time order.
obs_flat = observations.reshape(-1, 256) # [2000000, 256]
x_flat = sampled_x.reshape(-1) # [2000000]
logp_flat = saved_logp.reshape(-1) # [2000000]
adv_flat = advantage.reshape(-1) # [2000000]
returns_flat = returns.reshape(-1) # [2000000]
for _ in range(4):
for rows in torch.randperm(len(obs_flat)).split(25000):
ppo_minibatch(
obs_flat[rows], x_flat[rows], logp_flat[rows],
adv_flat[rows], returns_flat[rows],
)
Critic targets are computed before subtracting the group average. Only the actor’s advantages are centered across attempts.
Result
The final run trained for 5,000 epochs in about 10.4 hours on one RTX 5090. Its official score was 42.206 on 5,000 segments. PID scored 110.254 on the same evaluation. Lower is better. These segments came from the supplied training pool, so this measures the challenge result rather than performance on unseen roads.
Inference uses the trained actor and the mean of its steering distribution. It needs no critic or optimizer.
# One car. obs [1, 256], previous_steer [1].
with torch.no_grad():
dist = steering_distribution(obs)
x = dist.mean # alpha / (alpha + beta), shape [1]
delta = 0.25 * (2 * x - 1)
steering = (previous_steer + delta).clamp(-2, 2)
command = steering.item() # one Python number for the simulator