Is it wrong not to speak American or British English?

You don't need to speak British or American English, but you should certainly learn to recognize those dialects because they are the most commonly spoken and most easily understood.