I turned off settings that weren't needed ...
If you're concerned about privacy, check the apps and services that have permission to track your location.
Turns out that Google Maps is keeping track of more than your trips.