A physics-informed reinforcement learning-based strategy for local and coordinated ramp metering

Abstract

This paper proposes a physics-informed reinforcement learning(RL)-based ramp metering strategy, which trains the RL model using a combination of historic data and synthetic data generated from a traffic flow model. The optimal policy of the RL model is updated through an iterative training process, where in each iteration a new batch of historic data is collected and fed into the training data set. Such iterative training process can evaluate the control policy from reality rather than from a simulator, thus avoiding the RL model being trapped in an inaccurate training environment. The proposed strategy is applied to both local and coordinated ramp metering. Results from extensive microscopic simulation experiments demonstrate that the proposed strategy (i) significantly improves the traffic performance in terms of total time spent savings; (ii) outperforms classical feedback-based ramp metering strategies; and (iii) achieves higher improvements than an existing RL-based ramp metering strategy, which trains the RL model merely by a simulator. We also test the performance of two different learning algorithms in the simulation experiment, namely a conventional tabular approach and a batch-constrained deep RL approach. It is found that the deep RL approach is not as effective as the conventional tabular approach in the proposed strategy due to the limited amount of training data.