Dress level of shoes

As someone starting a shoe collection, I'm finding that with so many shades of brown, I would rather have the shades covered first, then flesh out with more styles. I am assuming that a dressier type of shoe is a better buy, such as an oxford, which (I assume) can be dressed down, than something more casual, which would look inappropriate with a suit. Is my thinking correct, or would oxfords look just as out of place when dressing more casually?

Right now I am looking for a dark brown shoe as a last purchase for some time, as I prefer to buy quality shoes less often. I have several shades of lighter and medium browns covered, but for some outfits, I really need something dark.

Thanks!

Dress level of shoes