I am so bummed that it took 31 years before I was made aware of Silver Lake.  It is closest to (or in, I guess) Kit Carson, Amador, California, near Kirkwood on 88.  What an amazing place for a camping trip.  We went on Labor Day weekend, which should have been the most crowded weekend of the year, and it still felt like we had the lake to ourselves.  

The fishing is decent, the swimming is excellent, but for those who hike, it's tough to beat.  We spent a full day just hiking the mountains around the lake, finding other little lakes in the higher areas and taking in some breathtaking views.  Based on that one day, I think it would take a good month of constant hiking to have covered all of the forests, hills, mountains, lakes, and caves surrounding Silver Lake.

Skip Yosemite for a year, and go to Silver Lake.  You won't regret it.