If you're short on time, it's easier -- and cheaper -- to visit without a car. Getting here by plane, floatplane, or bus and ferry is fairly seamless. Once here, you'll find most sights, restaurants, and hotels in the city's compact, walkable core, with Harbour ferries, horse-drawn carriages, double-decker buses, step-on tour buses, taxis, and pedicabs on hand to fill the gaps. For the more popular sights outside the core -- Butchart Gardens, Hatley Castle, the Scenic Marine Drive -- tour buses are your best bet. Taxis typically cost more than tours, and local public transit is geared more to commuters than sightseers.
A car, or bike, is handy (though not essential) if you're staying outside downtown in the Rockland or Oak Bay area. You will need a car if you're staying on the Saanich Peninsula or the Malahat, or if taking side trips to Sooke, the Cowichan Valley, or the Gulf Islands. Parking isn't especially expensive or hard to find in downtown Victoria, but parking enforcement is ruthless.
Biking is an excellent option in this, one of the most bike-friendly towns in North America. Bike paths lace downtown and run along much of the waterfront, long-haul car-free paths run to the ferry terminals and as far west as Sooke, and most buses and ferries carry bikes.
Visit the Travel Talk forums for help on planning your trip