In this work, we study the behavior of the case fatality rate (CFR) in a mixed population of vaccinated and unvaccinated individuals. We show that the evolution of the CFR depends on both the effectiveness of vaccines in preventing deaths and the detection of infections among both the vaccinated (breakthrough infections) and unvaccinated individuals on the CFR. This result suggests that a decline in the CFR may not imply that vaccines are effective in reducing deaths. Likewise, a constant CFR can still mean that vaccines are effective in reducing deaths. We conclude that unless efforts are directed at detecting breakthrough infections, it is hard to disentangle the effect of vaccines in reducing deaths from the probability of detecting infections on the CFR.