Control policies optimized in simulation can perform poorly in the real system when the simulator's parameters are estimated from limited data but the resulting parameter uncertainty is not represented inside the simulation. A common way to incorporate such ambiguity is to simulate each trajectory under a randomly drawn and unobservable parameter value. This forces the policy to act robustly at the start of the control episode. Over time, however, the policy can often infer the parameter value from its observations and specialize its actions accordingly, which is undesirable in systems where latent factors are expected to shift. In financial markets, for example, a policy hedging a derivative payoff should remain robust to changes in the volatility regime. To induce such continual robustness, we propose training policies in simulators where ambiguity can vary with the system's state but does not systematically vanish over time. We formalize this as stationary ambiguity: the simulator induces a stationary filter process over the latent state. We show how to construct such simulators and demonstrate, on hedging problems, that policies trained under stationary ambiguity maintain robustness to latent factors over time, leading to strong performance on real market data. As a modeling principle, stationary ambiguity informs which models make realistic simulators, how their parameters should be randomized, and how simulator and policy should be initialized. While our experiments focus on hedging, stationary ambiguity may also be useful for other control problems driven by exogenous stochastic processes with shifting latent structure.
This is a joint work with Amira Akkari, Ben Wood, and Lukas Gonon.