You can't brush suspicious graven in a spire area

So there is this artifact structure near the spire close enough it says spirit spire and shows how many life wisps you have. You can't brush the suspicious gravel in that area, only the suspicious gravel outside of that area.