Having lived and spent a lot of time in very large cities (NY, Tokyo, London, Washington), suburbs, and small towns, I know what you mean. I still remember how different NY was on my first visit. I now live in a village of 300 people that is mostly rural farms, but because of its relative proximity (75 miles) to a larger city, it 'feels' more urban than some cities with 75000 people in the midwest and west.
A few suggestions:
To get a visual feel for a city, try using Google maps in two ways. First, use the aerial view to start big and then zoom in. This will give you an idea of the sheer size and density of big cities, which is key to getting a feel for how and why they are different (and as someone else said, why in big cities people often tend to stick within enclaves). Second, use 'street view' to see what a street looks like. Much of Shanghai is like never ending high rises, yet NY and Los Angeles changes dramatically once you move outward. Get in a car in NY and in 25 minutes you can be out of the city; get on a train in Tokyo and in 90 minutes you are still in Tokyo...
Second, if you want to get a 'grittiness feel' for big cities, read something by Andrew Vachss, especially the early books (Flood, Strega). While he writes about NY, there are 'dark' places similar to what he describes in most big cities.
Third, remember you can't get it perfect with research only. Just as no one in San Francisco says "Frisco", all locales have their local knowledge that you will never get perfect unless you live there. (As just one example: if you get into a taxi at an airport in NY, locals will say "The city" to the driver, meaning "Take me to Manhattan" even though the NY airports are actually IN New York City.). One way to catch these mistakes/add local color is to have some beta readers from your described city setting.
Finally, don't sweat it. Even major writers make these mistakes all the time. I have a business background, and one of my pet peeves is how poorly business in general and companies in particular are described in the media and in fiction books. I'd say it is about 90% incorrect. Worse, even a cursory bit of research would correct these mistakes, but maybe the authors just find the generalizations (the 'evil' corporation, the three martini lunch, etc) to be useful to their stories).
Sometimes, you just have to let it go.