Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| en:docs:win16:modules:local_heap [2026/02/24 07:39] – [Arena Formats] prokushev | en:docs:win16:modules:local_heap [2026/02/25 09:10] (current) – prokushev | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ===== Win16 Local Heap Functions | + | {{page> |
| + | |||
| + | ====== DGROUP, | ||
| ===== Overview ===== | ===== Overview ===== | ||
| In 16-bit versions of Windows (1.x, 2.x, 3.x), each module (application or DLL) typically has its own data segment (DGROUP) limited to 64 KB. This segment contains the stack, the local heap, and the atom table. Local heap functions manage memory within this segment using near pointers (offsets). They are exported by the kernel (KRNL386.EXE). Internally, the local heap is managed through a set of data structures that reside inside the segment itself. | In 16-bit versions of Windows (1.x, 2.x, 3.x), each module (application or DLL) typically has its own data segment (DGROUP) limited to 64 KB. This segment contains the stack, the local heap, and the atom table. Local heap functions manage memory within this segment using near pointers (offsets). They are exported by the kernel (KRNL386.EXE). Internally, the local heap is managed through a set of data structures that reside inside the segment itself. | ||
| - | A key part of this management is the Instance Data (also called the NULL segment) located at the very beginning of DGROUP, which holds pointers to the heap, atom table, and stack information. The field at offset 6 (pLocalHeap) contains a near pointer to the HeapInfo structure that heads the local heap. This pointer is set by LocalInit() and must be validated by checking the signature word (li_sig) at offset 22h (Standard mode) or 28h (KRNL386) within the presumed heap. | + | A key part of this management is the Instance Data (also called the NULL segment) located at the very beginning of DGROUP, which holds pointers to the heap, atom table, and stack information. The field at offset 6 (pLocalHeap) contains a near pointer to the HeapInfo structure that heads the local heap. This pointer is set by [[en: |
| The local heap itself is organized as a series of arenas (headers) that precede each block. The two low bits of the la_prev field in an arena are used as flags: bit 0 indicates whether the block is in use (1) or free (0); bit 1 indicates whether the block is MOVEABLE (1) or FIXED (0). Free blocks are linked via la_free_prev and la_free_next. For MOVEABLE blocks, a separate handle table (pointed to by hi_htable in HeapInfo) stores the actual address, lock count, and flags; each handle is an offset into this table. | The local heap itself is organized as a series of arenas (headers) that precede each block. The two low bits of the la_prev field in an arena are used as flags: bit 0 indicates whether the block is in use (1) or free (0); bit 1 indicates whether the block is MOVEABLE (1) or FIXED (0). Free blocks are linked via la_free_prev and la_free_next. For MOVEABLE blocks, a separate handle table (pointed to by hi_htable in HeapInfo) stores the actual address, lock count, and flags; each handle is an offset into this table. | ||
| Line 11: | Line 13: | ||
| ==== Instance Data (NULL Segment) ==== | ==== Instance Data (NULL Segment) ==== | ||
| - | The first 16 (10h) bytes of the default data segment (DGROUP) are reserved for system use and are collectively | + | |
| + | The first 16 (10h) bytes of the default data segment (DGROUP) are reserved for system use and are referred to as the Instance Data or NULL segment. This area is present only if the first WORD at offset 0 is zero; otherwise, the structure is not present. The layout is as follows: | ||
| ^ Offset ^ Type ^ Field ^ Description ^ | ^ Offset ^ Type ^ Field ^ Description ^ | ||
| | 00h | WORD | wMustBeZero | Must be zero for the NULL segment structure to be considered present. | | | 00h | WORD | wMustBeZero | Must be zero for the NULL segment structure to be considered present. | | ||
| - | | 02h | DWORD | dwOldSSSP | When SwitchStackTo() is called, the current SS:SP is stored here. At other times, may contain the value 5 (from C compiler' | + | | 02h | DWORD | dwOldSSSP | When [[en: |
| - | | **06h** | **WORD** | **pLocalHeap** | Near pointer to the Local Heap information structure (i.e., the HeapInfo structure). This field is set by LocalInit() and points to the beginning of the local heap management structures. If no local heap exists, this field may be stale (non-zero but invalid). Always verify the heap by checking the signature at offset 22h (Standard mode) or 28h (KRNL386) – see li_sig below. | | + | | 06h | WORD | pLocalHeap | Near pointer to the Local Heap information structure (i.e., the HeapInfo structure). This field is set by [[en: |
| - | | **08h** | **WORD** | **pAtomTable** | Near pointer to the atom table structure. Set by InitAtomTable(). Zero until atoms are used. | | + | | 08h | WORD | pAtomTable | Near pointer to the atom table structure. Set by [[en: |
| | 0Ah | WORD | pStackTop | Near pointer to the end (top) of the stack. For DLLs, this is zero. | | | 0Ah | WORD | pStackTop | Near pointer to the end (top) of the stack. For DLLs, this is zero. | | ||
| | 0Ch | WORD | pStackMin | High‑water mark of stack usage. For DLLs, zero. | | | 0Ch | WORD | pStackMin | High‑water mark of stack usage. For DLLs, zero. | | ||
| Line 25: | Line 28: | ||
| * The field at offset 6 (pLocalHeap) is the primary way to locate the local heap structures given only the DGROUP selector. | * The field at offset 6 (pLocalHeap) is the primary way to locate the local heap structures given only the DGROUP selector. | ||
| - | * When LocalInit() is called on a globally allocated block (non‑DGROUP), | + | * When [[en: |
| - | * Similarly, if InitAtomTable() is called on a global block, offset 8 points to the atom table, and offset 6 will point to the associated local heap (since atoms are stored in the local heap). | + | * Similarly, if [[en: |
| ==== HeapInfo and LocalInfo ==== | ==== HeapInfo and LocalInfo ==== | ||
| + | |||
| Every local heap begins with an instance of the HeapInfo structure, which is identical to the one used by the global heap and is defined in WINKERN.INC. Its location is given by the pLocalHeap field at offset 6 of the Instance Data. Immediately following the HeapInfo structure are additional fields that, together with HeapInfo, form the LocalInfo structure. | Every local heap begins with an instance of the HeapInfo structure, which is identical to the one used by the global heap and is defined in WINKERN.INC. Its location is given by the pLocalHeap field at offset 6 of the Instance Data. Immediately following the HeapInfo structure are additional fields that, together with HeapInfo, form the LocalInfo structure. | ||
| ===== HeapInfo Structure (386) ===== | ===== HeapInfo Structure (386) ===== | ||
| + | |||
| In the enhanced mode (krnl386), the HeapInfo structure occupies **1Eh** bytes and has the following format (according to " | In the enhanced mode (krnl386), the HeapInfo structure occupies **1Eh** bytes and has the following format (according to " | ||
| ^ Offset ^ Type ^ Field ^ Description ^ | ^ Offset ^ Type ^ Field ^ Description ^ | ||
| | 00h | WORD | hi_check | If this value is nonzero, the debug version of KERNEL verifies the heap. This field appears to be used only for the local heap, not for the global heap. | | | 00h | WORD | hi_check | If this value is nonzero, the debug version of KERNEL verifies the heap. This field appears to be used only for the local heap, not for the global heap. | | ||
| - | | 02h | WORD | hi_freeze | If this is nonzero, KERNEL should not compact the heap. For the global heap, this value appears to be set only while inside the INT 24h handler. The local heap is frozen during LocalAlloc() and LocalRealloc(). | | + | | 02h | WORD | hi_freeze | If this is nonzero, KERNEL should not compact the heap. For the global heap, this value appears to be set only while inside the INT 24h handler. The local heap is frozen during |
| | 04h | WORD | hi_count | The total number of blocks in the heap. | | | 04h | WORD | hi_count | The total number of blocks in the heap. | | ||
| | 06h | DWORD | hi_first | A far pointer to the arena header for the first block in the heap. The first block is always a sentinel and points to itself. | | | 06h | DWORD | hi_first | A far pointer to the arena header for the first block in the heap. The first block is always a sentinel and points to itself. | | ||
| Line 70: | Line 75: | ||
| ===== LocalInfo Structure (386) ===== | ===== LocalInfo Structure (386) ===== | ||
| - | For **krnl386** (HeapInfo size of 1Eh), the LocalInfo structure has the following layout: | ||
| - | ^ Offset ^ Type ^ Field ^ Description ^ | + | For both **krnl386** and **krnl286**, the LocalInfo structure immediately follows the HeapInfo structure and has the following |
| - | | 00h | - | HeapInfo | 1Eh-byte HeapInfo structure as described above. | | + | |
| - | | 1Eh | DWORD | li_notify | Far pointer to a routine called either when a heap block is about to be moved or discarded, or when the heap is out of memory. Initialized to point at LocalNotifyDefault(). | | + | |
| - | | 22h | WORD | li_lock | Lock count of the local heap. A non‑zero value prevents blocks from moving or being discarded. | | + | |
| - | | 24h | WORD | li_extra | Minimum amount by which the local heap should be grown when expanded. Default is 200h. | | + | |
| - | | 26h | WORD | li_minsize | Minimum size of the local heap, as specified by the HEAPSIZE line in the .DEF file. | | + | |
| - | | 28h | WORD | li_sig | Signature word set to 484Ch (' | + | |
| - | + | ||
| - | ===== LocalInfo Structure (286) ===== | + | |
| - | For krnl286, the LocalInfo structure immediately follows the HeapInfo structure and contains additional fields that manage heap notifications, | + | |
| ^ Offset ^ Type ^ Field ^ Description ^ | ^ Offset ^ Type ^ Field ^ Description ^ | ||
| - | | 00h | - | HeapInfo | 16h-byte HeapInfo structure as described above. | | + | | 00h | DWORD | li_notify | Far pointer to a routine called |
| - | | 18h | DWORD | li_notify | Far pointer to a routine called when a heap block is about to be moved/discarded or when the heap is out of memory. Initialized to point to LocalNotifyDefault(). | | + | | 04h | WORD | li_lock | Lock count of the local heap. A non‑zero value prevents blocks from moving or being discarded. | |
| - | | 1Ch | WORD | li_lock | Lock count of the local heap. A non‑zero value prevents blocks from being moved or discarded. | | + | | 06h | WORD | li_extra | Minimum amount in bytes by which the local heap should be grown when expanded. Default is 200h. | |
| - | | 1Eh | WORD | li_extra | Minimum amount | + | | 08h | WORD | li_minsize | Minimum size of the local heap, as specified by the HEAPSIZE line in the .DEF file. | |
| - | | 20h | WORD | li_minsize | Minimum size of the local heap, as specified by the HEAPSIZE line in the .DEF file. | | + | | 0Ah | WORD | li_sig | Signature word set to 484Ch (' |
| - | | 22h | WORD | li_sig | Signature word set to 484Ch (' | + | |
| - | Important: | + | Important: |
| - | The total size of the local heap information block (HeapInfo + LocalInfo) for krnl286 is therefore 24h bytes (18h + 0Ch), though only the first 22h bytes are used up to the signature. The signature itself occupies bytes 22h–23h. | + | * In krnl286, the li_sig field is located at offset 22h from the beginning |
| + | * In krnl386, the li_sig field is located at offset 28h from the beginning of the combined HeapInfo + LocalInfo structure. | ||
| ==== Arena Formats ==== | ==== Arena Formats ==== | ||
| + | |||
| Every block in the local heap is preceded by an arena (header) that contains management information. Arenas always start on a 4‑byte boundary, so the two low bits of every arena address are zero. These bits are reused as flags in the la_prev field of each arena. The two low bits of la_prev have the following meaning: | Every block in the local heap is preceded by an arena (header) that contains management information. Arenas always start on a 4‑byte boundary, so the two low bits of every arena address are zero. These bits are reused as flags in the la_prev field of each arena. The two low bits of la_prev have the following meaning: | ||
| Line 106: | Line 102: | ||
| ===== FIXED Block Arena ===== | ===== FIXED Block Arena ===== | ||
| + | |||
| ^ Offset ^ Type ^ Field ^ Description ^ | ^ Offset ^ Type ^ Field ^ Description ^ | ||
| | 00h | WORD | la_prev | Near pointer to the preceding arena, with flags in the low two bits. | | | 00h | WORD | la_prev | Near pointer to the preceding arena, with flags in the low two bits. | | ||
| Line 113: | Line 110: | ||
| ===== MOVEABLE Block Arena ===== | ===== MOVEABLE Block Arena ===== | ||
| + | |||
| ^ Offset ^ Type ^ Field ^ Description ^ | ^ Offset ^ Type ^ Field ^ Description ^ | ||
| | 00h | WORD | la_prev | Near pointer to the preceding arena, with flags. | | | 00h | WORD | la_prev | Near pointer to the preceding arena, with flags. | | ||
| Line 121: | Line 119: | ||
| ===== Free Block Arena ===== | ===== Free Block Arena ===== | ||
| + | |||
| ^ Offset ^ Type ^ Field ^ Description ^ | ^ Offset ^ Type ^ Field ^ Description ^ | ||
| | 00h | WORD | la_prev | Near pointer to the preceding arena, with flags. | | | 00h | WORD | la_prev | Near pointer to the preceding arena, with flags. | | ||
| Line 131: | Line 130: | ||
| ==== The First Local Heap Block ==== | ==== The First Local Heap Block ==== | ||
| + | |||
| The first block in the local heap is special. It resides in memory before the LocalInfo structure. Although its la_prev field has the low bit set (indicating a FIXED, in‑use arena), the local heap routines treat this arena as if it were free. The la_free_next field of this first block points to the first truly free block in the heap. | The first block in the local heap is special. It resides in memory before the LocalInfo structure. Although its la_prev field has the low bit set (indicating a FIXED, in‑use arena), the local heap routines treat this arena as if it were free. The la_free_next field of this first block points to the first truly free block in the heap. | ||
| ==== Handle Table ==== | ==== Handle Table ==== | ||
| + | |||
| For MOVEABLE blocks, handles are offsets into a handle table that resides in its own local heap blocks. The first handle table is located via the hi_htable field in the HeapInfo structure. Each handle table begins with a WORD specifying how many handle entries follow. After all entries, the last WORD is the offset of the next handle table (or 0 if none). Free handle entries are linked together for quick allocation. | For MOVEABLE blocks, handles are offsets into a handle table that resides in its own local heap blocks. The first handle table is located via the hi_htable field in the HeapInfo structure. Each handle table begins with a WORD specifying how many handle entries follow. After all entries, the last WORD is the offset of the next handle table (or 0 if none). Free handle entries are linked together for quick allocation. | ||
| Line 149: | Line 150: | ||
| ===== Heap Operations ===== | ===== Heap Operations ===== | ||
| - | * **Allocation (LocalAlloc)** walks the free list, splitting blocks if necessary, and sets up the appropriate arena. For MOVEABLE blocks, it also allocates a handle table entry. | + | |
| - | * **Compaction (LocalCompact)** coalesces adjacent free blocks and may move or discard unlocked MOVEABLE blocks. When a block is moved, its lhe_address is updated. | + | * **Compaction (LocalCompact)** coalesces adjacent free blocks and may move or discard unlocked MOVEABLE blocks. When a block is moved, its lhe_address is updated. |
| - | * **Locking (LocalLock/ | + | * **Locking (LocalLock/ |
| - | * **Discarding (LocalDiscard)** frees the memory of a MOVEABLE block but keeps the handle entry alive with the LHE_DISCARDED flag set. | + | * **Discarding (LocalDiscard)** frees the memory of a MOVEABLE block but keeps the handle entry alive with the LHE_DISCARDED flag set. |
| ===== Atom Tables ===== | ===== Atom Tables ===== | ||
| Line 182: | Line 183: | ||
| String atoms are created by passing an ordinary string to `AddAtom` or `GlobalAddAtom`. They are stored in the atom table as `ATOMENTRY` structures. | String atoms are created by passing an ordinary string to `AddAtom` or `GlobalAddAtom`. They are stored in the atom table as `ATOMENTRY` structures. | ||
| - | * **Range**: `0xC000` to `0xFFFF` (encoded pointer). | + | |
| - | * **Storage**: | + | * **Storage**: |
| - | * **Reference count**: Yes (`usage` field). | + | * **Reference count**: Yes (`usage` field). |
| - | * **String representation**: | + | * **String representation**: |
| - | * **Creation**: | + | * **Creation**: |
| **Encoding: | **Encoding: | ||
| Line 199: | Line 200: | ||
| Integer atoms are created by passing a string of the form `"# | Integer atoms are created by passing a string of the form `"# | ||
| - | * **Range**: `0x0001` to `0xBFFF` . | + | |
| - | * **Storage**: | + | * **Storage**: |
| - | * **Reference count**: Not applicable. | + | * **Reference count**: Not applicable. |
| - | * **String representation**: | + | * **String representation**: |
| - | * **Creation**: | + | * **Creation**: |
| **How it works:** When a string of the form '# | **How it works:** When a string of the form '# | ||
| Line 217: | Line 218: | ||
| This macro casts a 16‑bit integer value to a pointer type. When this " | This macro casts a 16‑bit integer value to a pointer type. When this " | ||
| - | * For values ≤ `0xBFFF`, the function treats it as an integer atom and returns the value directly. | + | |
| - | * For values ≥ `0xC000`, the function assumes it is an encoded pointer to an `ATOMENTRY` and will dereference it (after shifting left by 2 bits) to access the atom entry. | + | * For values ≥ `0xC000`, the function assumes it is an encoded pointer to an `ATOMENTRY` and will dereference it (after shifting left by 2 bits) to access the atom entry. |
| **Important: | **Important: | ||
| Line 224: | Line 225: | ||
| ==== Local vs. Global Atom Tables ==== | ==== Local vs. Global Atom Tables ==== | ||
| - | * **Local atom tables**: Bound to a specific data segment (e.g., an application' | + | |
| - | * **Global atom table**: A system-wide table accessible to all applications via `GlobalAddAtom`, | + | * **Global atom table**: A system-wide table accessible to all applications via `GlobalAddAtom`, |
| ==== Creating Custom Atom Tables (outside DGROUP) ==== | ==== Creating Custom Atom Tables (outside DGROUP) ==== | ||
| Line 231: | Line 232: | ||
| Since all atom operations work with the current segment pointed to by DS, you can create and use an atom table in any arbitrary data segment by following three steps: | Since all atom operations work with the current segment pointed to by DS, you can create and use an atom table in any arbitrary data segment by following three steps: | ||
| - | 1. **Create a local heap** in the target segment using `LocalInit(Selector, | + | - |
| - | 2. **Switch the DS register** to that segment. | + | |
| - | 3. Call `InitAtomTable(size)` to initialize the atom table in the newly created heap. | + | |
| After that, any subsequent call to `AddAtom`, `FindAtom`, etc., will operate on the custom table if DS is temporarily set to the correct segment. | After that, any subsequent call to `AddAtom`, `FindAtom`, etc., will operate on the custom table if DS is temporarily set to the correct segment. | ||
| Line 261: | Line 262: | ||
| </ | </ | ||
| - | * `wSegment` – Selector of the segment where the heap will be created. | + | |
| - | * `pStart` – Offset of the first byte of the heap area (must be paragraph‑aligned, | + | * `pStart` – Offset of the first byte of the heap area (must be paragraph‑aligned, |
| - | * `pEnd` – Offset of the last byte of the heap area (inclusive). The heap will manage memory from `pStart` to `pEnd`. | + | * `pEnd` – Offset of the last byte of the heap area (inclusive). The heap will manage memory from `pStart` to `pEnd`. |
| If successful, `LocalInit()` returns a non‑zero value. It sets up the `HeapInfo` and `LocalInfo` structures at the beginning of the heap area (starting at `pStart`) and updates the segment’s instance data at offset **06h** (`pLocalHeap`) to point to that `HeapInfo` structure. However, if the segment is not a default data segment (i.e., not DGROUP), the instance data at offset 0 must also contain a zero word to indicate that the NULL segment structure is present; otherwise, the heap may not be recognized by some routines. | If successful, `LocalInit()` returns a non‑zero value. It sets up the `HeapInfo` and `LocalInfo` structures at the beginning of the heap area (starting at `pStart`) and updates the segment’s instance data at offset **06h** (`pLocalHeap`) to point to that `HeapInfo` structure. However, if the segment is not a default data segment (i.e., not DGROUP), the instance data at offset 0 must also contain a zero word to indicate that the NULL segment structure is present; otherwise, the heap may not be recognized by some routines. | ||
| Line 306: | Line 307: | ||
| **Important Considerations: | **Important Considerations: | ||
| - | * The heap structures themselves occupy space at the beginning of the heap area. The first block (sentinel) resides at `pStart + size of (LocalInfo)`. | + | |
| - | * The segment’s instance data (at offset 0) must be properly set up, especially the zero word at offset 0, to avoid confusion with other structures. | + | |
| - | * Custom local heaps are not automatically enlarged if they run out of space; they are limited to the range specified in `LocalInit`. | + | * The segment’s instance data (at offset 0) must be properly set up, especially the zero word at offset 0, to avoid confusion with other structures. |
| - | * The `HEAPSIZE` setting in the module’s .DEF file only affects the default DGROUP heap. | + | * Custom local heaps are not automatically enlarged if they run out of space; they are limited to the range specified in `LocalInit`. |
| + | * The `HEAPSIZE` setting in the module’s .DEF file only affects the default DGROUP heap. | ||
| ==== Creating Atom Tables Outside DGROUP ==== | ==== Creating Atom Tables Outside DGROUP ==== | ||
| Line 317: | Line 319: | ||
| ==== Summary of Custom Heap and Atom Table Creation ==== | ==== Summary of Custom Heap and Atom Table Creation ==== | ||
| - | * Use `LocalInit` on a segment to establish a local heap anywhere in memory. | + | |
| - | * The segment must have a valid NULL segment structure (zero word at offset 0) for the heap to be recognized. | + | * The segment must have a valid NULL segment structure (zero word at offset 0) for the heap to be recognized. |
| - | * After `LocalInit`, | + | * After `LocalInit`, |
| - | * To create an atom table in a custom heap, switch DS to that segment and call `InitAtomTable`. | + | * To create an atom table in a custom heap, switch DS to that segment and call [[en: |
| - | * All subsequent atom operations must be performed with DS set appropriately (or via wrapper functions). | + | * All subsequent atom operations must be performed with DS set appropriately (or via wrapper functions). |
| - | * Custom heaps and atom tables are useful for isolating memory pools, implementing resource managers, or working with large data structures without polluting the default DGROUP. | + | * Custom heaps and atom tables are useful for isolating memory pools, implementing resource managers, or working with large data structures without polluting the default DGROUP. |
| ===== References ===== | ===== References ===== | ||
| - | 1. Schulman, A., Maxey, D., Pietrek, M. // | + | - Schulman, A., Maxey, D., Pietrek, M. // |
| - | 2. Pietrek, M. //Windows Internals// | + | |
| - | 3. Chen, R. //The Old New Thing// (blog). Microsoft Developer Blogs. | + | |
| - | 4. Microsoft OS/2 Version 1.1 Programmer' | + | |




