I spent a few months in Vitoria and vicinity and fell in love with the city. Every travel guide I have read dismisses Vitoria as just a dangerous backwater with nothing of interest. I strongly disagree.
Vitoria is like a miniature Rio in terms of geography and scenery. It's set on a gorgeous backdrop of rolling hills that jut out in to the ocean, sandy beaches, and a large, sweeping bridge (the 3rd Point bridge) - the tallest in Brazil.
Vitoria as a city is booming, with high-end condos being built at a blistering pace, and shopping districts and restaurants popping up to serve the influx of new wealth. The old part of Vitoria (Vitoria Centro) is also undergoing a face lift with millions being poured in to renovation and preservation of old, historic buildings and museums. Roads are being re-surfaced and expanded, side walks are being repaired or re-built, and they are even planning a light-rail line to serve the city and it's environs (though Vitoria is served by a VERY comprehensive and efficient bus system).
Best of all, this small growing city is relatively unknown to tourists, and so tourist-related crimes are minimal. There is crime, but not more than any other city in Brazil. As long as proper precautions and common sense are followed, you should have a safe trip. Prices are not has high as other cities, and one can spend days exploring the museums, hills, and beaches in and around Vitoria.
Be prepared to bone up on your Portuguese, because very few know any English, and because tourists are so rare, you'll stick out no matter how hard you try to fit in. Still, if you notice people on the bus staring at you, it's likely because they are just curious, and not threatening. Many have never seen an Norte-Americano before, and will try to talk to you no matter how little Portuguese you know. Very friendly, helpful people.
So explore Vitoria before writing it off. I don't think a trip to Brazil would be complete without checking out this city. It's a real taste of what Brazil is WITHOUT all the tourist traps and glitz. Spend a few days and then judge for yourself. I don't think you'll be disappointed.