The Nightmare Realm is a warped, spooky version of the Invader Zim world that exists inside of Dib's mind. It is inhabited by Halloweenies, most of which are monstrous nightmare versions of regular characters. The ruler of the realm is Nightmare Bitters, the nightmare version of Ms. Bitters.
The only appearance and mention of this realm was in "Halloween Spectacular of Spooky Doom", in which an accident caused Dib to phase in and out between the normal world and the Nightmare Realm. Because the realm exists in Dib's mind, and contains horrible versions of nearly everyone in Dib's life, it can be assumed that the realm reflects an exaggerated image of how Dib sees the world.
Gallery
The home of Nightmare Membrane
Nightmare Membrane
The Nightmare Mech
Zim attempting to flee from the "Halloweens".
Nightmare Bitters