Anti-Exploration by Random Network Distillation
About
Despite the success of Random Network Distillation (RND) in various domains, it was shown as not discriminative enough to be used as an uncertainty estimator for penalizing out-of-distribution actions in offline reinforcement learning. In this paper, we revisit these results and show that, with a naive choice of conditioning for the RND prior, it becomes infeasible for the actor to effectively minimize the anti-exploration bonus and discriminativity is not an issue. We show that this limitation can be avoided with conditioning based on Feature-wise Linear Modulation (FiLM), resulting in a simple and efficient ensemble-free algorithm based on Soft Actor-Critic. We evaluate it on the D4RL benchmark, showing that it is capable of achieving performance comparable to ensemble-based methods and outperforming ensemble-free approaches by a wide margin.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Offline Reinforcement Learning | hopper medium | Normalized Score97.8 | 52 | |
| Offline Reinforcement Learning | walker2d medium | Normalized Score91.6 | 51 | |
| Offline Reinforcement Learning | walker2d medium-replay | Normalized Score88.7 | 50 | |
| Offline Reinforcement Learning | hopper medium-replay | Normalized Score100.5 | 44 | |
| Offline Reinforcement Learning | halfcheetah medium | Normalized Score66.6 | 43 | |
| Offline Reinforcement Learning | halfcheetah medium-replay | Normalized Score54.9 | 43 | |
| Offline Reinforcement Learning | D4RL antmaze-umaze (diverse) | Normalized Score66 | 40 | |
| Offline Reinforcement Learning | D4RL Adroit pen (human) | Normalized Return5.6 | 32 | |
| Offline Reinforcement Learning | D4RL Adroit pen (cloned) | Normalized Return2.5 | 32 | |
| Offline Reinforcement Learning | Walker2d medium-expert | Normalized Score105 | 31 |