Version control

Introduction

Version Control is a feature of the Creation Kit used by Bethesda developers to be to able to work on the same ESM file(s) simultaneously. Version Control is useful to modders because it can be used to merge the contents of ESP files into an ESM file. In the public version of the Creation Kit, Version Control is disabled by default. It can be enabled by following the instructions on this page.

You can also find Wheeze's Video Tutorial on Version Control here: http://youtu.be/6eInvAyMXoI

Setting up locally

These instructions are aimed at those who want to use Version Control merely for merging data of ESP files into a ESM file.

Create the following folders, assuming that Skyrim is your Skyrim installation directory:

Skyrim\Data\CheckInBackup
Skyrim\Merging\Data
Skyrim\Merging\VersionBackup

Make the folders CheckInBackup and Merging shared in your local network (can be a fake network with just your PC in it). Copy Skyrim.esm (and possibly Update.esm) and one or more of your own ESM files to Skyrim\Merging\Data. These are the "network versions". Add the following to SkyrimEditor.ini (some entries may already exist, watch out for them):

[General]
bUseVersionControl=1
SNetwork Path=\\Maegfaer-PC\Merging\
SNewVersionBackupPath=\\Maegfaer-PC\Merging\VersionBackup\
SNetworkMasterPath=\\Maegfaer-PC\Merging\Data\
SLocalSavePath=Saves\
SLocalMasterPath=Data\
SLocalBackupPath=Data\CheckInBackup

Replace Maegfaer-PC with the name of your computer.

Now go to SkyrimEditorPrefs.ini (the other file) and add the following to bottom:

[WhoCanMerge]
Maegfaer=1

Replace Maegfaer with your Windows User Account name.

Now start the CK, load Skyrim.esm (and possibly Update.esm) and your mod's ESM(s) by selecting file / Data files - or the yellow "folder" icon in the upper toolbar - and selecting them in the list of available mod files. While loading the files, the CK will probably show a warning dialog saying:

"DEFAULT: TESBitArrayFile::OpenAndMapBitArrayFile could not find '[modfile].fud'

This is expected - you can disable warnings.

When the mod files have finished loading, click on the "folder" icon again, select Skyrim.esm and click on the Details button. Answer with "Yes" when it asks to create an empty list. When the empty list shows, press 'Ctrl + Shift + B' and then press OK. Close and do the exact same for your mod's ESM(s). Now in your Skyrim\Merging\Data folders there are a number of files Version Control uses to merge ESP files into any of your ESM files.

If the "Empty List?" prompt doesn't come up, it's likely that there's something wrong with the username you have selected. There should be be a new file:

Skyrim\Merging\ConstructionSetNetwork.ini

containing (for example)

[Users]
1=Maegfaer
Maegfaer=1

Check that the username specified in here matches the one you specified in SkyrimEditorPrefs.ini. It may be different if your Windows user account has ever been renamed. The value created in ConstructionSetNetwork.ini is the one that the CK will use - copy it over to SkyrimEditorPrefs.ini.

Version Control is now ready to be used.

Using Version Control

Quick and Simple

Load the CK with your to-be-merged-into ESM file enabled and the ESP containing the data to be merged as Active. When the CK is done loading, click on the Version Control button to the far left of your upper toolbar. Check out all the forms you want to merge first, then select them again and press Check In. A new window pops up. Select the ESM you want to merge into. Choose No when it asks if you want to use a Check In Data file (CID) and there it goes, it merges the content of your ESP file into your chose ESM and resolves any formID conflicts.

Masters

When using Version Control, you can only have a maximum of 3 .esm files at once. During development, Bethesda's 3 .esm files were Skyrim.esm, Update.esm, and either Dragonborn.esm or Dawnguard.esm. This is likely a measure to promote stability as when the Creation Kit runs out of usable memory, it will crash. There are workarounds to this, but they aren't recommended.

Check Out/In

Remember that Version Control is a system made for a large group of developers building a world at the same time. Checking Out a formID means you "reserve" it, it gets "locked" in the Network master file (the ESM in Merging\Data) so that none modifies the same content at the same time, which would result in conflicts. Every dev always has a copy of the ESM on his local system (your regular Data folder). The reason why you need to Check Out (reserve it) before you Check In (merge it) is a part of this safeguard to prevent developers changing the same forms. These steps have little meaning if you only use Version Control on a local system. It is just how the system works.

Merging Multiple Plugins

The files with the extension .fvd, .fud etc stored in your shared network drive should be regenerated after each merge into an .esm file using the steps outlined above. Otherwise, the Creation Kit may not allow you to check out certain forms that have been altered in a plugin that touches records that have already been merged. To verify that you are not rewriting data that you had already merge or that you are not introducing injected records that have the potential to crash the Creation Kit, it is strongly recommended that you check each plugin for errors, injected records, dirty edits, and ITM records in TES5Edit.

CID files

When Checking In forms, the Creation Kit changes the formID's of those forms if necessary to prevent duplicated formID's in the ESM. It also changes the formID index to match the index of the ESM file. Any references in forms to other forms are adjusted to match the re-assigned formID's. On merging, all the re-assigned formID's get listed in a *.CID file in the SNewVersionBackupPath folder defined in the SkyrimEditor.ini.

CID files can come in handy when merging Plugin A and B into an ESM, while plugin B depends on plugin A. Let's assume plugin A is merged into the ESM first. Plugin A's forms get new formID's, which breaks any references in plugin B to forms originating from plugin A. When merging Plugin B into the ESM, the CID file created in the merging process of plugin A should be selected when asked for. That way all broken form references in plugin B get changed to the re-assigned forms of the data originating from plugin A.

Trick to check out Skyrim.esm forms

Checking out forms that belong to Skyrim.esm doesn't work by default. This can be problematic if you have your own ESM file and want to merge ESP files into your ESM. You won't be able to check out any Skyrim.esm forms that the ESP file modifies, and are thus unable to merge such changes.

The fix for this is to temporarily turn off the 'Use Localized Strings' flag in the Skyrim.esm header. You can do this by opening Skyrim.esm in a hex editor (such as frhed), and changing the ninth byte from 81 to 01. First make a backup of Skyrim.esm just in case this goes wrong!

The start of the file looks like this by default:

54 45 53 34      2C 00 00 00     81 00 00 00

Change it to:

54 45 53 34      2C 00 00 00     01 00 00 00

Save the change and load Skyrim.esm in the CK. You'll get a good number of errors about strings and such, ignore them and press Yes to All. Now it's possible to Check Out forms that belong to Skyrim.esm. Merge any changed Skyrim.esm forms from your ESP into your own ESM, then exit the Creation Kit and revert the flag change.

Merging navmeshes

In order to merge Navmesh forms (NAVM), the NavMeshInfoMap (NAVI) in Skyrim.esm must be checked out. Checking it out is an arduous process, but it generally only has to be done once on your setup and then it will remain checked out, even across merges, sessions and bit array regenerations.

Apply the aforementioned hex edit to Skyrim.esm.
Once in the version control dialog, click "All" and wait several minutes while a list of every loaded form is generated.
Sort by "Type", then find the NAVI form manually in the list.
Select it and check out. Then wait several more minutes while the list refreshes.
Go back to viewing Altered, then check out/check in all forms as usual.

Perforce Integration

Perforce is a 3rd Party Version Management software suite, available for free to small groups. When set up with the settings documented here, Perforce works hand-in-hand with the Version Control system to keep backups of previous iterations of scripts, and allow for each individual working on a project to pull the latest versions of all project scripts from the Perforce server. Perforce also manages DialogueView files, if present. In addition to storing versioned copies, the Creation Kit will update formids on all stored scripts and dialogue views that would otherwise change upon merge to an .esm.

With Perforce integration unlocked, it is possible for networked teams working off of a master server using just the Creation Kit and Perforce from a common server for version control.

Setup

First, set up Version Control per the above tutorial.

In SkyrimEditor.ini in your Skyrim folder, make sure that, under the [Papyrus] Settings, you have the following set as it is below:

sScriptSourceFolder = "Data\Scripts\Source"

sScriptCompiledFolder = "Data\Scripts\"

sCompilerFolder = "Papyrus Compiler\"

bPerforceEnabled = 1

In SkyrimEditorPrefs.ini make sure you have the below settings entered. Notes are in parentheses.

[Perforce]
sWorkspace=Papyrus (The name of your Perforce workspace - following this tutorial you will name it Papyrus)
sUsername=Rich (Your Perforce username - I've used the same as my Windows username)
[Papyrus]
sPerforceWorkspace=AWESOME-O (The name of your computer)
sPerforceUsername=Rich (Your Windows username)
sDLCPrefix=DLC (The prefix you wish to have added to any script fragments processed)

www.gameindustry.about.com/b/2012/07/29/version-control-management-and-games.htm

I will outline the steps roughly for now, but for step by step instructions please refer to that tutorial.

From http://www.perforce.com/downloads

Install Perforce Server using the suggested default settings including - and this is important - the default path. Make your username the same as your Windows username. You must have administrator permissions.

Set up your Perforce Depot to have the name TESVData and check that it is a stream depot rather than the default local depot type.

Next, install the Perforce Visual Client, also using default settings.

Open up your Perforce Server, ensure it is set up per the tutorial, then open P4V (the visual client). You will need to create a new workspace named Papyrus (not sure if this is hardcoded but it seems like it may be. Feel free to test other workspace names).

The workspace root should be in a new folder anywhere on your computer. I have mine in the folder above my Skyrim directory, since that is where the Source folder needed for LOD generation is kept. Don't add files yet if prompted.

In Windows Explorer, open up the new directory you created and add the following folders:

\current\Source\scripts \current\Source\DialogueViews \scripts

Copy all of the contents of your Skyrim\Data\Scripts\Source folder into the \current\Source\scripts subfolder. This should be thousands of .psc files and the subfolders for each DLC. If you are using a custom prefix for your project, you will need to set it in your Creation Kit preferences as. Do not rename script fragments manually, even through the Creation Kit.

Refresh your Perforce Client and you should see the new subfolders and files. Right click on the scripts folder and click add. It should put your files into a changelist. Then, right click again and click submit to commit those changes to the depot. You will need to add a comment, it can be anything.

Next, install the Microsoft Office plugin for Perforce. This package contains P4COM.dll, a necessary file for the Creation Kit to "talk" to Perforce as well as several other libraries/dlls. If you have a copy of P4Com.dll or P4com64.dll in your Skyrim directory already, you will need to manually register it with Windows.

Then restart your computer.

You should now be able to load up the Creation Kit and go to your preferences. At the very end is a tab for Perforce integration, where you can select your new workspace. Make sure your username is correct, select it, and hit apply.

To verify that you have a fully functioning install, when you open up the Version Control menu on the far left, the Papyrus Scripts box will no longer be grayed out. Instead, you should have a white screen with the Select All button clickable and the Check In/Undo Check Out Buttons grayed out.

However, the above isn't quite enough to confirm a correct install. As long as Perforce is present and P4COM.dll is registered the above will look right. To verify that the Creation Kit is able to read your TESVData depot, navigate to the Gameplay > Papyrus Script Manager option on the Creation Kit toolbar and click open. It should take a few seconds to load up. For most scripts in the manager, it should say "Yes" in both the "Compiled" and "Has Source" columns. The manager looks like that all the time (if you've properly extracted scripts.rar to the right folder) but if P4COM.dll isn't registered or something else has gone wrong, the Creation Kit won't be able to find the source scripts.

Troubleshooting

Errors

Failed: Check in canceled. Failed to access check in lock file.

Your shared network folders may have read permission, but not write permission.

Old editor warning: Your 'CreationKit.exe' is old and you must update from the network before check in is allowed.

You may have the wrong username set in SkyrimEditorPrefs.ini under [WhoCanMerge]. Double check whether it is the name which appears in the generated ConstructionSetNetwork.ini.

MASTERFILE: File index # is invalid. Clamping to ##. Does a master file depend on more than one master?

The above is a serious error resulting from having an active .esp with more than three master .esm files (including Skyrim.esm). It will corrupt both the .esm file you are merging into as well as the .esp file you are trying to merge if ignored. It is possible to bypass this error, allowing for one or more additional master files as long as all master files from position 02 or lower have no overlapping forms. The best way to do this is to manually renumber the forms in your fourth master file to a higher formid range, such as XX600000+ or XX7000000+ in TES5Edit prior to development.

Unable to initialize PerForce, check your logs for more information.

Perforce integration is not set up. It indicates that the Creation Kit will fall back on default code to assign new formids to script fragments, deleting the original versions without creating backups. It can be safely ignored, but be sure to keep manual back-ups.

Perforce workspace does not contain a script folder, Papyrus scripts will not be checked in or out.

You do not have a scripts folder in the root of your workspace. This is not the same folder as the source scripts - its purpose isn't clear but it may be a staging ground for formid updates.

Out of date target file: Only up to date master files can be checked in to. File '<x>.esm' has been checked into since it was loaded. Exit and update then try again.

Occurs when trying to check in if the esm changed between loading it, and the check in attempt. Can have several causes:

You made an edit in the CK.
You already did a merge in the same session.
The CK automatically fixed an issue in your esm on load. One such issue is null records, which it will remove. If this happens, load the CK with just your master file, close it, and copy the changed esm from your Merging to your local data folder. Then load it anew. A telltale sign that this is happening is if you get a warning indicating that your master file differs in the local and network folders.

Checkout Warning: <Form> is a different version than it was when it was loaded. Check out prevented.

You need to regenerate the bit array (see above). You may have done a merge earlier and haven't regenerated the bit array since.