The short answer is that if you are young and have a robust immune system, swimming east of the Harvard Bridge, without recent rainfall, you should be fine. I have done it on hot summer days and it was beautiful. The government, which plays these things pretty conservatively, agrees that on most days, the river basin meets the standards for swimmability below the Mass. Ave. bridge.
The longer answer is that the conditions vary depending on where you are in the river and the day. For centuries, the Charles was practically Boston's sewer, not just for human waste but also for all manner of industrial heavy metals.
In 2010, most sites east of Magazine Beach would have been "swimmable" about 75% of the time. See this presentation last October by the Massachusetts Water Resources Authority. (http://www.charlesriverc
Unfortunately, last year we had a problem with "Harmful Algae Blooms," perhaps caused by hot water runoff from power plants. The state posted an algae advisory from July through September. See http://www.charlesriverco
During the summer, the Charles River Watershed Association maintains a system of weekly monitoring and flies flags to show whether the water is "safe for boating" at nine places. You can get the data at http://www.crwa.org/water
As I understand, the standard for "safe for boating" is less than 630 colony-forming-units of E. coli per 100 mL, plus an acceptable level of blue-green algae. The EPA standard for swimming is tighter, I understand around 225 colony-forming units, plus a limit on Enterococcus.