The weight of the batteries is one aspect. Another aspect is the continuous variable transmission (CVT), which isn't great for circuit use. Also, when used intensively, the batteries easily run out of power. The concept works best for front wheel driven cars, as most braking power is on those wheels. All wheel drive would be an alternative, but less fun.
My last car was a Prius 3, which I drove for about 100,000 miles. The concept is great for all day use, but don't expect the direct response required for fast lap times, drifting and other fun. The car was extremely reliable and pretty quick from 0 to 50 mph.