I haven't worked out how to remove markers either.

I had assumed right clicking on it, but nope. No combination of alt/ctrl/shift or LMB/MMB/RMB would get rid of it.