The three guards who stand watch outside Lord Bafford's Manor are sometimes referred to as Bafford Front Gate Guards. When Garrett first approached Lord Bafford's Manor, they could be heard discussing the Bear Pits. (T1: "Lord Bafford's Manor")
These guards are indistinguishable from regular Bafford Guards, with the exception of two key points: their health is 25 points, rather than the usual 15, and they never flee from combat. In addition, they are slightly faster than the average Bafford guard. The guards will ignore Garrett unless he comes at them attacking, or if he approaches them and doesn't leave when they order him to do so.
Attacking them is quite difficult partly because of their higher than usual health, but also because their team includes an archer who can hit from a distance while the other two attack with swords. The best course of action is to avoid these guards entirely, which is quite easy to do.