Eu usei o código abaixo:

 AlertDialog.Builder bld;

 if (android.os.Build.VERSION.SDK_INT <= 10) {
     //With default theme looks perfect:
     bld = new AlertDialog.Builder(AndroidLauncher.this);
 } else {
     //With Holo theme appears the double Dialog:
     bld = new AlertDialog.Builder(AndroidLauncher.this, android.R.style.Theme_Holo_Dialog_MinWidth);
 }

 bld.setIcon(R.drawable.ic_launcher);
 bld.setTitle("Exit");
 bld.setMessage("Are you sure you want to exit?");
 bld.setNegativeButton("Cancel", new DialogInterface.OnClickListener() {
     @Override
     public void onClick(DialogInterface dialog, int which) { dialog.dismiss(); }
 });
 bld.setPositiveButton("Exit", new DialogInterface.OnClickListener() {
     @Override
     public void onClick(DialogInterface dialog, int which) { finish(); }
 });
 bld.setCancelable(false);
 bld.create().show();

Parece bom, mas diz "importar android.app.AlertDialog não pode resolver". É um projeto libGDX padrão no Android Studio.