Chọn phương án B.

\(\begin{eqnarray*}
&\lim\dfrac{n+\sqrt{n^2+1}}{\sqrt{n^2-n-2}}&=a\cdot\sin\dfrac{\pi}{4}+b\\
\Leftrightarrow&\lim\dfrac{n+\sqrt{n^2\left(1+\dfrac{1}{n^2}\right)}}{\sqrt{n^2\left(1-\dfrac{1}{n}-\dfrac{2}{n^2}\right)}}&=\dfrac{a\sqrt{2}}{2}+b\\
\Leftrightarrow&\lim\dfrac{n+n\sqrt{1+\dfrac{1}{n^2}}}{n\sqrt{1-\dfrac{1}{n}-\dfrac{2}{n^2}}}&=\dfrac{a\sqrt{2}}{2}+b\\
\Leftrightarrow&\lim\dfrac{1+\sqrt{1+\dfrac{1}{n^2}}}{\sqrt{1-\dfrac{1}{n}-\dfrac{2}{n^2}}}&=\dfrac{a\sqrt{2}}{2}+b\\
\Leftrightarrow&\dfrac{1+\sqrt{1+0}}{\sqrt{1-0-0}}&=\dfrac{a\sqrt{2}}{2}+b\\
\Leftrightarrow&2&=\dfrac{a\sqrt{2}}{2}+b.
\end{eqnarray*}\)

Vì \(a,\,b\in\mathbb{Z}\) nên suy ra \(\begin{cases}
a=0\\ b=2.
\end{cases}\)

Vậy \(S=0^3+2^3=8\).