We compute analytically the matter-dependent contributions to the quartic Casimir term of the four-loop lightlike cusp anomalous dimension in QCD, with nf fermion and ns scalar flavors. The result is extracted from the double pole of a scalar form factor. We adopt a new strategy for the choice of master integrals with simple analytic and infrared properties, which significantly simplifies our calculation. To this end, we first identify a set of integrals for which the integrands have a d log form, and are hence expected to have uniform transcendental weight. We then perform a systematic analysis of the soft and collinear regions of loop integration and build linear combinations of integrals with a simpler infrared pole structure. In this way, only integrals with ten or fewer propagators are needed for obtaining the cusp anomalous dimension. These integrals are then computed via the method of differential equations through the addition of an auxiliary scale. Combining our result with that of a parallel paper, we obtain the complete nf dependence of the four-loop cusp anomalous dimension in QCD. Finally, using known numerical results for the gluonic contributions, we obtain an improved numerical prediction for the cusp anomalous dimension in N=4 super Yang-Mills theory.