\begin{align*} F(b) - F(a) & = \int_{-\infty}^b f(x) dx - \int_{-\infty}^a f(x) dx\\ & = \int_a^b f(x) dx \\ & = P(a \lt x \le b) \end{align*}