Getting a system setting means a query that is executed. In scenarios where this happens a lot (think of an API Wrapper service that gets an API key and the API root URL and the API call timeout all via 3 systemsettings) - and the process is in a loop over a couple of thousand records (in my case a mass sync of data running in a task manager task) this could mean thousands of not wanted queries. AFAIK query caching is turned off for task manager task processing.
What if the SystemConfigurationService would have a whole caching layer included? System settings change very infrequently. It could be required to have a new interceptor that invalidates a specific setting category on insert/update. That way you can always make sure the correct values are in the cache. Otherwise (if you rely on the SystemConfigurationService itself only, you are fine to invalidate caches when saveSetting or deleteSetting is used. (I would favor the more straight forward approach to just have a cache within that service and invalidate on save/delete there.