The amount of combat xp you get varies depending on a couple of factors. Here's the formula for calculating combat xp in Java:
int xp = (int) (Math.round((percentage / 100) * 10) + (killstreak-2) * 5);
int playersonline = Bukkit.getOnlinePlayers().size();
if (playersonline <= 4) xp = (int) (xp * 0.25);
if (playersonline >= 5 && playersonline <= 9) xp = (int) (xp * 0.50);
//the variable xp is how much xp you get
//playersonline is the amount of players online
//killstreak is the player's current killstreak
Basically for every 10% of damage you deal to a player you will get +1 combat xp on kill and you will get +5*(your current killstreak) combat xp after reaching a triple kill. Then if there is less than 10 players you will get 50% of the xp, if there's less than 5 you will only get 25%.
If there is 10 players online and you deal 100% of the damage to the player you will get 10 combat xp. If you dealt the same damage with less than 10 players you would get 5 xp, less than 5 players will give 2 xp. Let's say you are on a penta kill (killstreak = 5), if you killed a player and dealt 100% of the damage and would get 30 xp (10+(6-2)*5) if there is 10 players online.
The best way to get the most xp is to reach high killstreaks and deal most of the damage.