When you click on a primary attribute, the product image will change even if you have not selected the full combination of current variations. To keep the default variation swatches, please select the "None" variant in the list.