The classical action for pure Yang-Mills gauge theory can be formulated as a deformation of the topological BF theory where, beside the two-form field B, one has to add one extra-field m given by a one-form which transforms as the difference of two connections. The ensuing action functional gives a theory that is both classically and quantistically equivalent to the original Yang-Mills theory. In order to prove such an equivalence, it is shown that the dependency on the field m can be gauged away completely. This gives rise to a field theory that, for this reason, can be considered as semi-topological or topological in some but not all the fields of the theory. The symmetry group involved in this theory is an affine extension of the tangent gauge group acting on the tangent bundle of the space of connections. A mathematical analysis of this group action and of the relevant BRST complex is discussed in detail.