ĐKXĐ: \(\sin x \ne 0\); \(\cos x \ne 0\) và \(\tan x \ne - 1\).
Ta có: \(\cot x = \dfrac{1}{{\tan x}}\);
\(\begin{array}{l}\cos 2x = 2{\cos ^2}x - 1\\ = 2\dfrac{1}{{{{\tan }^2}x + 1}} - 1\\ = \dfrac{{1 - {{\tan }^2}x}}{{{{\tan }^2}x + 1}}\end{array}\);
\(\begin{array}{l}{\sin ^2}x = 1 - {\cos ^2}x\\ = 1 - \dfrac{1}{{{{\tan }^2}x + 1}} = \dfrac{{{{\tan }^2}x}}{{{{\tan }^2}x + 1}}\end{array}\);
\(\begin{array}{l} - \dfrac{1}{2}\sin 2x = - \sin x\cos x\\ = - \dfrac{{\sin x}}{{\cos x}}{\cos ^2}x = - \tan x\dfrac{1}{{{{\tan }^2}x + 1}}\end{array}\)
Phương trình \(\cot x - 1 \)
\(=\dfrac{{\cos 2x}}{{1 + \tan x}} + {\sin ^2}x - \dfrac{1}{2}\sin 2x\)
\( \Leftrightarrow \dfrac{1}{{\tan x}} - 1 \)
\(=\dfrac{{\dfrac{{1 - {{\tan }^2}x}}{{{{\tan }^2}x + 1}}}}{{1 + \tan x}} + \dfrac{{{{\tan }^2}x}}{{{{\tan }^2}x + 1}} - \dfrac{{\mathop{\rm \tan x}\nolimits} }{{{{\tan }^2}x + 1}}\)
Đặt \(t = \tan x\) ta được \(\dfrac{1}{t} - 1 = \dfrac{{\dfrac{{1 - {{\mathop{\rm t}\nolimits} ^2}}}{{{{\mathop{\rm t}\nolimits} ^2} + 1}}}}{{1 + {\mathop{\rm t}\nolimits} }} + \dfrac{{{{\mathop{\rm t}\nolimits} ^2}}}{{{{\mathop{\rm t}\nolimits} ^2} + 1}} - \dfrac{{\mathop{\rm t}\nolimits} }{{{{\mathop{\rm t}\nolimits} ^2} + 1}}\)
\( \Leftrightarrow \dfrac{1}{t} - 1 = \dfrac{{1 - t}}{{{t^2} + 1}} + \dfrac{{{t^2} - t}}{{{t^2} + 1}}\)
\( \Leftrightarrow \dfrac{{1 - t}}{t} = \dfrac{{1 - t}}{{{t^2} + 1}} + \dfrac{{t(t - 1)}}{{{t^2} + 1}}\)
\( \Leftrightarrow \left[ \begin{array}{l}1 - t = 0\\\dfrac{1}{t} = \dfrac{1}{{{t^2} + 1}} - \dfrac{t}{{{t^2} + 1}}\end{array} \right.\)
\( \Leftrightarrow \left[ \begin{array}{l}t = 1\\{t^2} + 1 = (1 - t)t\end{array} \right.\)
\( \Leftrightarrow \left[ \begin{array}{l}t = 1\\2{t^2} - t + 1 = 0\text{(vô nghiệm)}\end{array} \right.\)
\(\begin{array}{l}t = 1 \Leftrightarrow \tan x = 1\\ \Leftrightarrow x = \dfrac{\pi }{4} + k\pi \in \mathbb{Z}\text{(thỏa mãn)}\end{array}\)
Vậy phương trình có nghiệm là \(x = \dfrac{\pi }{4} + k\pi \in \mathbb{Z}\).