Talk:CreateBoolArray - Utility

Bool array fill bug

A coding error in SKSE means that the contents of Bool arrays are effectively random. SKSE and SKSE64 both have the same defect, which arises from how Papyrus APIs added by SKSE (and DLLs based on SKSE's code) return array values to the game engine.

An SKSE-specific data structure called VMResultArray<T> (SKSE / SKSE64), where T can be substituted out for any data type, is used to more easily build arrays, and then this class is translated into a Skyrim-specific VMArray.

VMResultArray<T> is a customized version of std::vector<T>, a standard C++ data type for resizable lists. To "pack" the contents into a VMArray, SKSE loops over the contents using an iterator: a standard C++ wrapper object (think of a candy wrapper) that offers a unified way to loop over the contents of a variety of different lists, no matter how those lists arrange their contents in memory. To access the list item that an iterator wraps around, you need to "dereference" (unwrap) it by writing *myIterator. For each item in the array, then, SKSE dereferences an iterator to get the list item, takes the memory address (pointer) of the result, and casts that to a pointer of the appropriate type (just for good measure, I guess) before passing that pointer into a function that "packs" a single value into a VMArray. For most array types, this works fine: dereferencing the iterator gives you a value of the appropriate type; taking its address gives you a pointer of the appropriate type; the cast is probably redundant; and the code to "pack" a single value receives exactly what it expects to receive.

Unfortunately, the C++ standard contains a footgun. Usually, vectors store their elements sequentially in memory, one after the other, exactly as those elements are normally encoded when writing code. However, std::vector<bool> is not stored the same way. It is a "possibly space-efficient specialization" and the way it stores its values is up to individual compiler developers (so, Microsoft, in this case). Typically, a bool value is stored as an entire byte, even though only one bit is actually meaningful, because it's just easier for a CPU to work with it that way. A std::vector<bool> may instead pack up to eight bools into each byte, and do a little extra work to actually pull individual bools out and work with them. This means that std::vector<bool> has to use a unique iterator. This iterator doesn't point directly to a single bool; instead, it constructs a special "proxy object" that wraps around the bool. The contents of the proxy object are also up to whoever developed the compiler, but I would expect it to consist of a pointer to a vector and the index of a bool in that vector.

Here's what happens when SKSE tries to return a Bool array. It iterates over the VMResultArray<bool>. It dereferences the iterator and gets what it assumes is a bool; however, it's actually a proxy object that wraps around a bool. SKSE then takes a pointer to the proxy object, incorrectly casts that to a bool pointer, and passes that to the function which "packs" a single value into a VMArray. That code therefore misreads the data in the proxy object and treats that data like a bool. It basically ends up stuffing literal nonsense into the Bool array that SKSE wants to return.

This bug prevents the fill parameter in Utility.CreateBoolArray from working properly. However, it also has a much further-reaching effect: SKSE DLLs that use SKSE's codebase cannot create any Papyrus APIs that return Bool[]. Any such APIs will have their return values corrupted en route to the game, resulting in them returning arrays that have the correct length but are filled with nonsensical (usually True) values. DavidJCobb (talk) 20:06, 31 August 2022 (EDT)

CommonLibSSE specifically handles the vector-of-bool case in its code for packing arrays (Github / backup), so DLLs built with it should be able to add Papyrus APIs that return Bool[] properly. DavidJCobb (talk) 20:15, 31 August 2022 (EDT)