This issue came up several years ago and
again in 2020. In the 2nd & 3rd edition rules, the condition was, "Charging cavalry." So the consensus then was that only a charging cavalry unit was requited. Since infantry and cavalry didn't often operate together, mixing the 2 in assault didn't happen very often.
Now we have the 4th edition rules. Now the condition is written as, "Attacking units are charging cavalry." Implicitly that would be all the attacking units are charging cavalry. So if playing RaW, my interpretation would lean toward "all".
I don't play RaW, and in this case, I go with 2nd/3rd edition consensus. Whichever way you go, just be aware of what you are doing, and try to resolve these nuances with your opponent before you start playing.