Sunderland's Brian Brobbey should have been sent off for two yellow-card offences against Tottenham, a Premier League panel has ruled.