The guards found in Lord Bafford's Manor, often referred to as Bafford guards, are relatively sloppy and slow guards found throughout the level. The guards carry either a sword or a bow, and deal the same damage as Garrett's sword and bow. These guards are mainly found in Thief: The Dark Project/Thief Gold - "Lord Bafford's Manor". There are some missions (e.g. Assassins and Thieves' Guild) in which the Bafford Guards are used as street patrol and won't react upon spotting Garrett, unless being threatened.
Lord Bafford's guards are not very alert and are located sparsely around the mansion, so avoiding them should not be difficult. They are slow and imprecise and only moderately aggressive. It should not be difficult to lose them.