Taking you own items from a building and chests you made is fine, and also fine even if the house is no longer yours but the chests are still protected with your name.
Griefing your own building in your own town is your own prerogative .. however griefing a building in a town you left even if it was a building you made is greifing and lets be frank something people only do out of spite.. so don't be that guy if that is what has happened here.
And seriously stop the sillyness both of you.