Only cleanup rails temporary directory if it exists