I fixed it by deleting a layer file manually. Maybe it is because of the special character in the labels name
_Cave. Though renaming it did not solve the crash, only deleting it would.
<?xml version="1.0" encoding="UTF-8" ?>
<save>
<header version="2" time="1506965807" />
<version major="3" minor="1" revision="3" build="1" />
<region id="MetaData">
<node id="root">
<attribute id="ID" value="8a1282d4-40b9-78d2-de81-680c28d1f7e7" type="31" />
<attribute id="Name" value="_Cave" type="23" />
</node>
</region>
</save>