Version them in their with their timestamp. This way, when they change, the link will change and the browser will reload the sitemap… Otherwise it will be fine to be cached as it wont have changed.