From Nielson Norman Group: https://www.nngroup.com/articles/interpreting-research-findings/
Summary: If your product looks good from one perspective and bad from another, you have to check the methodology and try to interpret the findings.
While presenting a recent training seminar on quantitative research for UX , I was asked an interesting question:
“I’m leading a team tasked with overhauling a complex enterprise product . We have a high-fidelity prototype, and we’ve been conducting extensive research to get it ready for launch. The stakes are high — even minor improvements could lead to big productivity gains, but conversely, minor issues in the design could cause big problems for our users.
In our quantitative usability testing, we’ve seen substantial reductions in the amount of time it takes people to do important tasks in the prototype as compared to the old product. But here’s the problem — in qualitative interviews, our users hate the prototype . The feedback is so negative. How do we reconcile this contradiction? What should we do?”