Right clicking a custom map marker should delete it.