Se eu usar o JDK1.8_40 ou mais recente (o Oracle ou o OpenJDK fazem o mesmo), o código a seguir, juntamente com um redimensionamento da caixa de diálogo, travará o aplicativo (tentei o JDK do Windows 7, x64, 64 bits)

import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JPanel;
import javax.swing.JWindow;
import javax.swing.SwingUtilities;
import javax.swing.Timer;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;

public class Main {

public static void main(String[] args) {
    SwingUtilities.invokeLater(new Runnable() {
        @Override
        public void run() {
            final JDialog dialog = new JDialog();
            dialog.add(new JPanel());
            dialog.setVisible(true);
            dialog.setBounds(100, 100, 100, 100);

            final JWindow dependentWindow = getjWindow(dialog);
            dependentWindow.setVisible(true);
            dependentWindow.setBounds(100, 100, 100, 100);
            Timer t = new Timer(300, new ActionListener() {
                @Override
                public void actionPerformed(ActionEvent e) {
                    dependentWindow.setVisible(!dependentWindow.isVisible());
                }
            });
            t.start();
        }
    });
}

private static JWindow getjWindow(JDialog dialog) {
    JWindow w = new JWindow(dialog);
    JPanel panel = new JPanel();
    panel.add(new JButton("button"));
    w.add(panel);
    return w;
}
}

Não encontrei outras queixas sobre isso e ainda não postei um bug no site da oracle. Uma possível solução alternativa é alterar o JWindow para um JDialog não decorado, mas isso vem com outros problemas para mim, então eu não mudaria isso ainda.

Mais alguém encontrou esse problema e encontrou uma solução alternativa?

Adicionada a pilha:

WARN 2015-05-04 15:21:21,707 - AWT-EventQueue-0, Id = 17, Priority = 6: RUNNABLE
sun.awt.windows.WWindowPeer.reshapeFrame(Native Method)
sun.awt.windows.WDialogPeer.reshape(Unknown Source)
sun.awt.windows.WComponentPeer.setBounds(Unknown Source)
sun.awt.windows.WWindowPeer.setBounds(Unknown Source)
java.awt.Component.reshapeNativePeer(Unknown Source)
java.awt.Component.reshape(Unknown Source)
java.awt.Window.reshape(Unknown Source)
java.awt.Component.setBounds(Unknown Source)
java.awt.Window.setBounds(Unknown Source)
java.awt.Component.resize(Unknown Source)
java.awt.Component.setSize(Unknown Source)
java.awt.Window.setSize(Unknown Source)

Detalhes do problema do Windows (mostra 2 erros):

Problem signature:
Problem Event Name: BEX64
Application Name:   java.exe
Application Version:    8.0.60.13
Application Timestamp:  55404a69
Fault Module Name:  StackHash_08b3
Fault Module Version:   0.0.0.0
Fault Module Timestamp: 00000000
Exception Offset:   0000000300000002
Exception Code: c0000005
Exception Data: 0000000000000008
OS Version: 6.1.7601.2.1.0.256.48
Locale ID:  1033
Additional Information 1:   08b3
Additional Information 2:   08b36dcca93c38acb7c92ef4a729e798
Additional Information 3:   5d68
Additional Information 4:   5d682eddcc7a5d6b5452fc95535d5ac9

o segundo:

Problem signature:
Problem Event Name: APPCRASH
Application Name:   java.exe
Application Version:    8.0.60.13
Application Timestamp:  55404a69
Fault Module Name:  StackHash_d693
Fault Module Version:   0.0.0.0
Fault Module Timestamp: 00000000
Exception Code: c000041d
Exception Offset:   0000000300000002
OS Version: 6.1.7601.2.1.0.256.48
Locale ID:  1033
Additional Information 1:   d693
Additional Information 2:   d6933f192f50114566e03a88a59a6417
Additional Information 3:   9096
Additional Information 4:   9096dfe271c183defc2620e74bdaec28