We introduce Go-With-Uncertainty (GowU), a new approach to exploration in reinforcement learning that treats exploration as a particle-based search guided by uncertainty, rather than as learning a policy to maximize an exploration objective. GowU achieves state-of-the-art results on Montezuma’s Revenge, Pitfall!, and Venture, and solves pixel-based MuJoCo Adroit and AntMaze tasks without expert demonstrations..