HOW TO KICK OUT A MEMORY MANAGER
Warning : this article is meant for coders that are already experienced
enough with system programming. Sorry, but the concepts we'll have to use
are complicated enough to prevent me from explaining everything from the
very start. I think that this article will be long enough like that.
If you read this coding corner since its first issue (it was in Imphobia #7)
you should already know that one of my occupations was trying to find a way
to kick out the memory managers that likes to disable the use of the flat
real mode, or more generally to disable the acces to code privilege ring #0
(CPL0). Well, here is it.
The LOADALL option was not the good one. It is not supported on all types of
CPU. For example my intel 486DX2 does not support it. Moreover this instruct
seems to be a priviledged one, so it was impossible to use it to access to
CPL0.
However there must be a way to access to CPL0 with a memory manager loaded.
Do you think that Micro$oft Windows runs in enhanced 386 mode with a CPL3 ?
I don't. So there was only one possible solution : they used some kind of
undocumented interface to gain total control to the CPU.
This interface is known as the "Windows Global EMM Import Specification" and
we will just write GEMMIS in the rest of this text.
Here I need to thank my friend Cedric BERMOND aka LCA / Infiny for his great
help. He pointed out to me the Micro$oft Windows argument, and he told me
about the following references. He implemented his own system at the same
time as me, and we had a lot of chats about the problems we encountered and
the solutions we found. Without him the development of this system would
have been impossible. The code example included with this article is written
by me, but it's the result of a collaboration between us.
LCA will soon (before the Assembly'95) release his own protected mode memory
manager, featuring :
* user/supervisor protection system, with the user running at CPL0
* own linker, for a true flat segmentation model (for example the screen
really is at A0000 as opposed to pmode)
* paging disabled by default
* full debugger and disassembler system
Some references about the GEMMIS :
* Ralf Brown's interrupt list. Provides some quite detailed information
about the undocumented functions and structures that we will use, but with
no explanations at all. When I read this at the first time I believed that
this functions were a part of the Windows kernel and I skipped the rest. I
was wrong, this is some undocumented functions made FOR windows to allow
him to access to CPL0
* Dr Dobb's Journal, September 1994, Undocumented Corner,
"The Windows Global EMM Import Interface". This article provides some ex-
planations and gives the general layout we will have to use, but with a
lack of detail.
This problem becomes even more important today that the organizers of The
Assembly 95 said in a pre-invitation textfile that at this party the demos
will have to run with EMM386.EXE loaded. This sucks badly, however in this
article you have the solution. The only annoying thing is that I guess that
the organizer's reason is not a technical one, it is just because they want
to sell their damn CD's.... :(
Okay, now let's start with the serious things.
PART ONE - THE LAYOUT
We have two basic needs :
* switching to real, unprotected mode
* allocate some high memory
To satisfy this needs we will have to use the following layout :
* call int 2f undocumented function 1605h
"Micro$oft Windows - WINDOWS ENHANCED MODE & 286 DOSX INIT BROADCAST"
This call will fool the memory manager and he will believe that we are
Windows trying to initialize itself. The memory manager will suddenly become
very friendly with us. He will also provide us a pointer to a function that
we can call to switch to real, unprotected mode.
* before the switch we will allocate all available XMS and EMS memory
* In order to be able to use the allocated memory blocks we must know their
physical addresses. With the XMS it's easy, we just have to 'lock' them.
With the EMS it's harder because we have no documented way to know about
which physical pages are allocated to our EMS handle. That's no problem, the
switch to real mode function will return this information in a table. We
just need to know the physical address where this table will be, and we need
to use another undocumented function for that : we must do an IOCTL read on
the "EMMXXXX0" device, and the read will return the desired address along
with the version of the GEMMIS protocol. This function is referenced in RB's
interrupt list as "Memory Managers - GET EMM IMPORT STRUCTURE ADDRESS"
* then we use the switch to real mode function that was provided to us at
the first step. It fills us a table with all the information we could need
about the memory mapping and the pages allocated to our EMS handle, then it
gives us acces to the real mode.
* we need to analyse this table. It is located in high memory , so we will
have to switch to flat real mode (or in protected mode if you prefer it) in
order to access this table. It won't be a big problem : now we are in real
mode and the memory manager is not there to annoy us. But we must take care
of not enabling the interrupts : the UMB's and EMS page frames does not
exist anymore, and an interrupt would most likely hang the system.
* after that we have several ways to cope with the interrupts : the simplest
one is to be sure to NEVER call the any system interrupt during your demo,
the medium one is to switch back to V86 mode before any interrupt (it's easy
to do, we just have to use the same function that allowed us to acces to
real mode), and the hardest one (the one used by Micro$oft Windows) is to
code your own memory manager that will be able to run the system with the
same mapping that the DOS-memory manager used.
* When we have finished playing with the computer we must give it back to
the system. We just have to switch it back to V86 mode.
* Then we must call int 2f undocumented function 1606h
"Micro$oft Windows - WINDOWS ENHANCED MODE & 286 DOSX EXIT BROADCAST"
This will return the memory manager to his normal, unfriendly state.
PART TWO - THE DETAILS
* first step : call int 2f undocumented function 1605h "Micro$oft Windows -
WINDOWS ENHANCED MODE & 286 DOSX INIT BROADCAST" at the entry of this call
we must set ax=1605h (function number), and bx=cx=dx=si=ds=es=0. di will be
set of the Windows version number we want to simulate. For example we can
use 30ah for Windows version 3.10 then we call int 2fh. If the memory mana-
ger accepts to run windows he will return cx=0, in any other case we will
have to exit. The memory manager will return in ds:si a far pointer to a
modeswitch routine, used later to switch between V86 and real mode. It can
also return 0:0, and in this case we won't be able to enter real mode. The
memory manager will create a EMMXXXX0 device if there is not already one.
* then we have to allocate the memory. The simplest way is to allocate XMS
memory, because we can then use the documented XMS function 0ch to lock the
memory block and get his address. We want to allocate ALL the XMS memory and
perhaps it will already be fragmentated, so we should be able to handle
several XMS blocks. However some memory managers won't allow us to allocate
all the memory using XMS allocations, or sometimes they won't even support
the XMS standard. So we must also allocate some EMS memory. If we want to be
100% sure of our memory allocations we can use standard XMS & EMS functions
to copy a known string in all our allocated memory blocks. Later we will be
able to verify that this string is present at the expected physical addres-
ses. This provides us a way to verify the physical adresses we will use. In
order to later identify our EMS handle it is best to assign it a name.
* we must get the GEMMIS address with an IOCTL read on the "EMMXXXX0" device
"Memory Managers - GET EMM IMPORT STRUCTURE ADDRESS" we will first verify
that there is an EMM present, using the standard method defined in lotus/
intel/microsoft Expanded Memory Specification (EMS) version 4.0 : we must
open a handle for the EMMXXXX0 device, using the same function as for a file
open, i.e. a call to int 21h with ax=3d00h and ds:dx pointing to the device
name. If the handle exists (that should be the case) we issue a 'get device
information' IOCTL, i.e. a call to int 21h with ax=4400h and the handle to
the EMMXXXX0 device in bx. This call will return us some information about
our handle in dx, and we must verify that bits 14 and 7 are set, indicating
that this handle is associated to a device driver and not a file, and that
this device driver accepts IOCTL reads. We will then issue a 'get output
status' IOCTL on this device, i.e. a call to int 21h with ax=4407h and the
device handle in bx. we expect to have ax=255 upon exit, indicating that the
device is ready. At this step we know that there is an EMM in memory, but we
don't know which version of the EMM specification does it support. We close
our device handle (using standard file close function ah=3eh bx=device
handle) and we ask for EMM version, using int 67h function ah=46h. The
version returned in al should be at least 40h i.e. 4.0 now this memory mana-
ger should be recent enough to support the GEMMIS specification, so we will
attempt an GEMMIS IOCTL read on it. We must open again the EMMXXXX0 device
(same process as above) and issue an IOCTL read on it i.e. a call to int 21h
with ax=4402h bx=the device handle ds:dx pointing on a 6 bytes buffer and
cx=6=number of bytes to read. The first byte of the buffer must be pre-
filled with a 01h value. (it's used by the GEMMIS protocol as a sub-function
value) upon exit we should have AX=6=nb bytes read, and the buffer will be
filled with a dword value indicating the physical memory address of the
GEMMIS data table, and a word value indicating the version of the GEMMIS
specification. (format for the version word : one byte for the major number,
and one byte for the minor number). We expect the version to be at least 1.0
(the current version is 1.11 but is not supported by every manager. Version
1.0 will be enough to know the pages allocated to our EMS block) then we can
close again the device driver.
* we want to switch to real mode. We will use the modeswitch function
provided at the first step of the GEMMIS procedure. if we call it with ax=0
it will switch us to real mode. The carry flag should be cleared upon exit
to indicate a success. This function can (and will) destroy every register
except CS,IP,SS and SP. the DS,ES,FS,GS segments registers will also be des-
troyed. The switch to real mode will of course destroy the UMB's and EMS
frames, so we take care not to enable the interrupts after this step. We
must clear the interrupt flag before the call, and we can expect it to be
still cleared after the call. This function will not only switch us to real
mode, it will also fill up the GEMMIS data block with the V86 memory mapping
using a data format that I won't detail here, but you can look at the
GEMMIS.DOC file that I spread you with my code example, it's an extract from
RB's interrupt list and it's very detailed.
* Now we run in real mode (at CPL0) so we can do what we want with the CPU.
It's time to initialize flat real mode, or protected mode at CPL0 or whate-
ver else we wanted to have acces to. We must just have access to all the
physical memory, because the GEMMIS data table will be located in high memo-
ry and we won't be able to acces it from [non-flat] real mode. Nothing spe-
cial to explain here, I'll assume that you already know how to enter into
flat real mode.
* Now that we have access to all the physical memory it's time to analyse
the the GEMMIS data table. Just look at its format in the GEMMIS.DOC file,
and at my tips in the next section. This analysis will allow us to get the
physical addresses of the pages of our EMS handle. The interesting part of
the GEMMIS data block will be the EMS handle info records, however we will
need to use the rest of the GEMMIS data block to find them. We musn't rely
on the EMS handle number to identify our EMS handle, it is necessary to use
the EMS handle name we assigned. (because 386max won't give us the right
handle numbers).
* We have done with the GEMMIS protocol. The only problem now is that we
can't enable interrupts, because the interrupt handlers could try to access
to an UMB or to map some EMS pages and they don't exist anymore. So we can
either keep the interrupts disabled, or create some new interrupt handlers
that will switch back the CPU to V86 mode (using the modeswitch function),
call the old interrupt handler, and switch the CPU to flat real mode again.
(or to protected mode if you prefer, do what you like)
* At the end of our program (or when an interrupt occurs) we will want to
switch back the CPU to V86 mode, in order to restore the UMB and EMS native
structure of the DOS system. We just have to call the modeswitch function
with AX=1. Again, this call will destroy every register except CS,IP,SS,SP.
and we must call it with the interrupts disabled.
* At the end of our code we will have to fake a Windows exit. We just have
to call int 2fh with ax=1606h and dx=0
Phew, we are done with this shitty GEMMIS protocol ! Yes, but there is more
to come.....
PART THREE - THE PROBLEMS
Here I have to explain that this interface was keept very secret. There was
no official documentation about it and some of the memory manager makers did
not had the specification in the hands when they had to support it. For
example it is known that Novell had to reverse engineer some code to support
the last version of the specification. This explains why the GEMMIS protocol
is so badly implemented in several memory managers. Some data records are
sometimes not filled for example.... :(
By the way I think it shows us a very bad attitude from Micro$oft. They
sometimes are accusated to monopolyse the market but I just have to say that
this is true, because I cannot admit that someone keeps secret a specifica-
tion that is necessary for a memory manager to support Windows, and for a
Windows-like program to support the memory managers. It's a shame.
So here is a few tips that you must use if you want your GEMMIS implementa-
tion to support every memory manager around (you should read this part of
the article with a printed copy of GEMMIS.DOC near you. This is only imple-
mentation details, so you won't understand anything if you don't have the
format of the GEMMIS data table in the hands)
* at offset 4 in the GEMMIS data you should find a word containing the
GEMMIS version number. Don't use it. Several memory managers, for example
QEMM386 v6.0, won't fill it. Instead you should use the version number
provided by the GEMMIS IOCTL read, at offset 4 in the 6-bytes import buffer.
* don't rely on the informations provided by GEMMIS version 1.10 : several
memory managers still only implement version 1.0 (for example 386max)
* in the EMS frame status records, don't use the "flags for non-EMS frames"
byte (located at offset 5) : several memory managers, including 386max and
QEMM386, won't fill it. Use the "EMS frame type" at offset 0 instead.
* use the "number of UMB frame descriptors following" byte at offset 18Bh in
the GEMMIS data. Sometimes, for example if there is 386max loaded, the UMB
frame descriptor table will not be entirely filled but this byte will always
have the right value.
* the "number of EMS handle info records following" appears to be reliable.
* in each EMS handle info record, ignore the "handle number" byte. Better
use the "EMS handle's name" to identify your EMS handle. This is necessary
with 386max. The "physical address of page table entries forming page map"
entry points to a table giving us the physical memory adresses used for each
4K-page of the EMS handle. This table contains 4*"number of 16K pages for
handle" double words, and each or this double words gives us the physical
address of one 4K-page used for the EMS handle. The lower 12 bits of this
double words should be ignored because they will be filled with some
garbage.
* once again : don't use the information provided by GEMMIS version 1.10,
even Micro$oft's EMM386 v4.49 does not seems to fill it... You can just know
the structure in order to skip them, thus giving you access to the GEMMIS
version 1.11 records (the memory manager name)
* when you allocate your XMS memory, you will probably want to use the XMS
function 08h "Query Free Extended Memory". At this step don't rely on
getting an errorlevel in BL. Every XMS function returns some errorlevels in
BL, and zero means OK, but obviously Micro$oft just forgot the line saying
"BL = 00h if the function succeeds" in their original "eXtended Memory
Specification (XMS), ver 3.0" publication. The programmers at QuarterDeck
probably are not intelligent enough to understand this evidence, so when
everything is OK QEMM386 does not modify the BL register. So you should set
BL to zero before the call, and also test for AX=0 (no available XMS memory)
instead of just relying on the errorlevel in BL.
* QEMM is harder to support than the others. With QEMM you will have to
restart the whole protocol if you want to temporarily switch back to V86
mode. Here is the thing you will have to code if you want to switch back to
V86 mode for a little time (for launching a system interrupt for example) :
disable flat real mode, switch to native V86 mode using the modeswitch func-
tion, fake a windows exit, call your system interrupt, fake another windows
init, switch back to real mode with the modeswitch function, enable flat
real mode again. You may think it's stupid to fake a windows exit just for
entering windows again later, yes it's stupid but remember : we don't try to
be smart, we just try to emulate Windows code. ;-)
PART FOUR - THE IMPLEMENTATION
As an example I'll show you a simple flat real mode initializer. This imple-
mentation supports RAW, XMS and EMS memory allocations. It's a "do-nothing"
implementation, i.e. it just initializes flat real mode, then restores the
previous mode and exits to dos. Well this should be enough for demonstration
purposes.
It's written in turbo assembler 2.0, and I compile it with "tasm gemmis" and
"tlink /m/3 gemmis". I guess it could also be compiled with further versions
of turbo assembler, but I didn't tested this.
It should be very easy to use this code : just include what you want between
the init and end portions of the code, it could be a flat real mode program
or a protected mode initialization with CPL0, and it will work. You can use
the 'available_memory_table', it will give you the adresses of every alloca-
ted memory block. The number of such blocks will be stored in 'nb_available_
memory'
The init of this code may seem slow, the main reason for this is that I fil-
led every allocated memory block with a known string in order to verify that
the physical adresses are the right ones after the flat real mode switch.
There is two standards provided for interrupt support: The easyest one is to
just call the interrupt, there will be a handler installed which will trap
the interrupt and switch back to native V86 mode to execute it. Then the
system will return to flat real mode and you will not have to worry about
the modeswitch. However it will be a bit slow. (by the way you should trap
the timer interrupt yourself if you want to use this code in a demonstration
in order to avoid all this unnecessary mode switches)
The other method for the interrupt support is to ask the memory manager to
disable the flat real mode in a code section delimited by two function
calls. In this section you won't be in flat real mode, but you will be able
to call every interrupt without any unnecessary delay. I call that a native
mode code section.
Look at my code example : you just have to call mem32_init to setup every-
thing. At exit you will be in flat real mode and all the free high memory
will be allocated to you. You will be able to use the data in
available_memory_table to get some memory : it contains the starting and
ending physical adresses of every allocated memory blocks. The number of
such blocks will be stored in nb_available_memory.
When you want to define a native mode code section you'll just have to call
the mem32_native and mem32_flatreal functions.
When you want to exit your proggy just jump to the exit label, with DS:DX
pointing to a message you want to display.
As an example proggy I just included a proggy that dumps the GEMMIS v1.0
data table. It's damn slow because I use DOS interrupts to display the
information and this causes a lot of mode switches. In a real code you
probably would like to build your own string displayer and it would be
pretty much faster.
This implementation has been successfully tested with the following memory
managers : HIMEM.SYS alone, EMM386.EXE v4.49, QEMM386.SYS v6.00 beta,
386MAX.SYS v6.02
PART FIVE - GOING FURTHER
In my code example I used flat real mode, but it could be easy to change the
library to use protected mode if you prefer it. By the way a lot of protec-
ted mode coders insists on the fact that protected mode is faster because it
eliminates the need for prefixes before 32-bit instructs, but do they know
that it's also possible to use some 32-bit flat real mode without this ugly
prefixes ??? most of them does not seems to.
The obvious problem with this little example is that you have a delay each
time you want to call an interrupt. Because of that you will want to make
most of your init work in each part in a native mode code section, without
the benefits of privilege level 0, and then start your part without any call
to a V86 interrupt.
The alternate method, a much better one but far beyond the scope of this
introduction, would be to have your own V86 mode handler which would be able
to run the native system interrupts. This way you could bypass all the open
windows/close windows shit. If someone goes this way, I would be very happy
to get a little words from him. This is the way microsoft windows handles
the whole thing.
PART SIX - GOODBYE
Demomakers : the only thing you have to remember is that if this article and
the associated code example is useful for you I would be happy to get a lit-
tle greet. It would be nice also to say hello to my friend LCA/Infiny. If
you encounter some incompatibility problem with some hardware or with some
memory manager please let us know. Now I'm more or less obliged to include
the following disclaimer :
You can use this code example as you wish if it's for your private use. I
would prefer to receive a letter just to know that someone uses it, but it's
not necessary. Do what you want.
Here's an adress if you want to join me :
Walken / IMPACT Studios [coder]
Michel LESPINASSE
18 rue Jean Giono
80090 Amiens
France