User:DavidJCobb/TESVSnip

I spent a bit of time looking into TESVSnip to better understand what the historical issues with it were, solely because I got nerd-sniped by something semi-similar. I didn't come up with anything I'd consider a satisfying answer, not least because C# isn't my cup of tea.

So take everything below with a grain of salt.

From what I can see, TESVSnip began as a Fallout 3 tool in the Fallout Mod Manager. Someone named Figment forked it into its own program meant for use on Skyrim; this, I believe, may have been the version that people saw produce broken plug-ins. Some time later, Sharlikran (better known for contributing to xEdit) took over TESVSnip in an attempt to repair it, but this proved to be a dead end.

I've looked over some claims made about TESVSnip, and I've looked over its code. I can't find any specific evidence of where the claimed breakages definitely would occur. I'm not saying that TESVSnip is a totally functional program whose name has been unfairly maligned over the years; I was just unable to find answers for myself in the few hours I wasted on this (and I don't intend to waste any more of them).

Other people's testimony

This old forum post from June 7, 2012 is the most complete record I've found, and states:

One flag, 0x00040000, indicates that the Field list is compressed. In this case the "data" area contains a four-byte "size of decompressed data" and the compressed stream. We discovered that sometimes the actual decompressed data is longer than the size indicated. The extra data do not appear to be more Fields. We had thought that an undocumented flag - 0x02000000 - was a warning of this but that turned out to be unreliable. TESVGecko will retain the entire decompressed data stream even though we don't know what it is for.

If a plugin utility drops this extra data, it is gone from the plugin. Opening the plugin in the Creation Kit will not necessarily restore this information. Also, if the compression flag is set, then the data must be compressed. Having the compression flag set but providing uncompressed data will cause issues, likely crashing the game or the Creation Kit. We will not be flipping this flag in TESVGecko. If the flag is set, then the data will be recompressed when the plugin is saved.

[...]

UPDATE: The issues we found earlier were due to using the SharpZipLib port of zLib. This port does not include all of the zLib library. You must use zLib v. 1.2.3 to decompress/compress Elder Scrolls plugin data properly. This applies to all versions of Elder Scrolls and Fallout games up to and including Skyrim.

[...]

Summary of issues:

- TESVSnip uses the stated size of decompressed data and does not process the actual size of the decompressed data, thus truncating the extra data

- TESVSnip does not flip the compression flag when it decompresses the data and saves the plugin. This will cause data corruption because the Creation Kit is being told the data are compressed, but the actual data are not.

Someone put out a fix for TESVSnip which will recompress the data upon save, but the truncation issue has not been fixed.

And these issues are caused by the compression library TESVSnip uses. TESVGecko originally used the same library and we discovered that data were being corrupted when we attempted to re-compress the data upon save. We have now switched to the zLib library.

My thoughts:

TESVSnip uses the stated size of decompressed data and does not process the actual size of the decompressed data, thus truncating the extra data

The reason the compressed datastream begins with the size of the decompressed data is because if you're writing a program that wants to read this data, you need to know how much memory to allocate to hold the decompressed data. You can't predict that in advance because the compression ratio isn't fixed; not all data will be compressed to the exact same proportion of its original size, across the board. If the size isn't declared (or isn't declared properly), then you have to reallocate a properly-sized buffer and try again (zlib's uncompress function should tell you the size you need).

Compressed records in well-formed ESP files should always declare the correct decompressed size. My own tool, DovahKit, aborts file load with a hard error if a declared decompressed size is too small, and I can load the latest versions of all official LE files just fine.

It does appear that TESVSnip relied entirely on the declared decompressed size and didn't validate that the data was fully decompressed (so, not even an error). If a record did declare an insufficient size, TESVSnip's older attempts at supporting compressed records would silently truncate the data. Sharlikran's attempted update would fail in the same way, I believe, beginning by allocating a fixed-size buffer sized to the declared decompressed size, and decompressing into it here, without anything I can see that would handle the case of the declared decompressed size being too small.

TESVSnip does not flip the compression flag when it decompresses the data and saves the plugin. This will cause data corruption because the Creation Kit is being told the data are compressed, but the actual data are not.

This would be a devastating issue; at minimum, the record declaring a serialized size larger than its actual size would cause it to "swallow" bytes from the next record or group, basically breaking the entire file. As far as I can see, however, TESVSnip has updated that record flag on save ever since Figment's Jan 18, 2012 commit that added the ability to save compressed records (see below). Additionally, the record header is written to state the compressed size plus sizeof(uint32_t); the record body is written as a uint32_t declaring the decompressed size, followed by the compressed data. Granted C# isn't my forte, but I genuinely can't spot where things might be going wrong.

The source code

As far as I can tell, the history of TESSnip's code goes like this:

• https://github.com/foesmm/fomm/tree/0.13.21
• https://github.com/figment/tesvsnip
• http://skyrim-plugin-decoding-project.googlecode.com/svn@258 (defunct)
• https://github.com/TESnip-Skyrim-Edition/TESV_Snip

Looking at the latest repo, I see that record flags are read here; if the record's contents are compressed, they get zlib-decompressed but the "compressed" flag remains set. The "compressed" flag is later updated at save time, and the record's size is the size of the compressed data. We can also refer to the Jan 18, 2012 commit in Figment's fork that added the ability to save compressed records in the first place, and see how it saves compressed records. That commit writes the decompressed size as the total serialized size of all subrecords.

Looking further back, at the FOMM TESSnip code, we can see that the code for saving records had no provisions in place for saving compressed data. However, they did decompress data on load and clear the "compressed" record flag, so prior to supporting the ability to save compressed data, they should always have been able to save decompressed record data back out.

As far as I can tell, the code for reading a record has always used the record's declared decompressed size as an input to decompress. I haven't looked through every commit but as far as I can tell, TESSnip has always used the declared decompressed size when reading records (i.e. the record header size is treated as the size of the compressed data; the compressed data begins with a declaration of the decompressed size; TESSnip honors both).