Difference between revisions of "Version control"

Revision as of 13:41, 13 January 2014

Introduction

Version Control is a feature of the Creation Kit used by Bethesda developers to be to able to work on the same ESM file(s) simultaneously. Version Control is useful to modders because it can be used to merge the contents of ESP files into an ESM file. In the public version of the Creation Kit, Version Control is disabled by default. It can be enabled by following the instructions on this page.

You can also find Wheeze's Video Tutorial on Version Control here: http://youtu.be/6eInvAyMXoI

Setting up locally

These instructions are aimed at those who want to use Version Control merely for merging data of ESP files into a ESM file.

Create the following folders, assuming that Skyrim is your Skyrim installation directory:

Skyrim\Data\CheckInBackup
Skyrim\Merging\Data
Skyrim\Merging\VersionBackup

Make the folders CheckInBackup and Merging shared in your local network (can be a fake network with just your PC in it). Copy Skyrim.esm (and possibly Update.esm) and one or more of your own ESM files to Skyrim\Merging\Data. These are the "network versions". Add the following to SkyrimEditor.ini (some entries may already exist, watch out for them):

[General]
bUseVersionControl=1
SNetwork Path=\\Maegfaer-PC\Merging\
SNewVersionBackupPath=\\Maegfaer-PC\Merging\VersionBackup\
SNetworkMasterPath=\\Maegfaer-PC\Merging\Data\
SLocalSavePath=Saves\
SLocalMasterPath=Data\
SLocalBackupPath=Data\CheckInBackup

Replace Maegfaer-PC with the name of your computer.

Now go to SkyrimEditorPrefs.ini (the other file) and add the following to bottom:

[WhoCanMerge]
Maegfaer=1

Replace Maegfaer with your Windows User Account name.

Now start the CK, load Skyrim.esm (and possibly Update.esm) and your mod's ESM(s) by selecting file / Data files - or the yellow "folder" icon in the upper toolbar - and selecting them in the list of available mod files. While loading the files, the CK will probably show a warning dialog saying:

"DEFAULT: TESBitArrayFile::OpenAndMapBitArrayFile could not find '[modfile].fud'

This is expected - you can disable warnings.

When the mod files have finished loading, click on the "folder" icon again, select Skyrim.esm and click on the Details button. Answer with "Yes" when it asks to create an empty list. When the empty list shows, press 'Ctrl + Shift + B' and then press OK. Close and do the exact same for your mod's ESM(s). Now in your Skyrim\Merging\Data folders there are a number of files Version Control uses to merge ESP files into any of your ESM files.

If the "Empty List?" prompt doesn't come up, it's likely that there's something wrong with the username you have selected. There should be be a new file:

Skyrim\Merging\ConstructionSetNetwork.ini

containing (for example)

[Users]
1=Maegfaer
Maegfaer=1

Check that the username specified in here matches the one you specified in SkyrimEditorPrefs.ini. It may be different if your Windows user account has ever been renamed. The value created in ConstructionSetNetwork.ini is the one that the CK will use - copy it over to SkyrimEditorPrefs.ini.

Version Control is now ready to be used.

Using Version Control

Quick and Simple

Load the CK with your to-be-merged-into ESM file enabled and the ESP containing the data to be merged as Active. When the CK is done loading, click on the Version Control button to the far left of your upper toolbar. Check out all the forms you want to merge first, then select them again and press Check In. A new window pops up. Select the ESM you want to merge into. Choose No when it asks if you want to use a Check In Data file (CID) and there it goes, it merges the content of your ESP file into your chose ESM and resolves any formID conflicts.

Check Out/In

Remember that Version Control is a system made for a large group of developers building a world at the same time. Checking Out a formID means you "reserve" it, it gets "locked" in the Network master file (the ESM in Merging\Data) so that none modifies the same content at the same time, which would result in conflicts. Every dev always has a copy of the ESM on his local system (your regular Data folder). The reason why you need to Check Out (reserve it) before you Check In (merge it) is a part of this safeguard to prevent developers changing the same forms. These steps have little meaning if you only use Version Control on a local system. It is just how the system works.

CID files

When Checking In forms, the Creation Kit changes the formID's of those forms if necessary to prevent duplicated formID's in the ESM. It also changes the formID index to match the index of the ESM file. Any references in forms to other forms are adjusted to match the re-assigned formID's. On merging, all the re-assigned formID's get listed in a *.CID file in the SNewVersionBackupPath folder defined in the SkyrimEditor.ini. CID files can come in handy when merging Plugin A and B into an ESM, while plugin B depends on plugin A. Let's assume plugin A is merged into the ESM first. Plugin A's forms get new formID's, which breaks any references in plugin B to forms originating from plugin A. When merging Plugin B into the ESM, the CID file created in the merging process of plugin A should be selected when asked for. That way all broken form references in plugin B get changed to the re-assigned forms of the data originating from plugin A.

Trick to check out Skyrim.esm forms

Checking out forms that belong to Skyrim.esm doesn't work by default. This can be problematic if you have your own ESM file and want to merge ESP files into your ESM. You won't be able to check out any Skyrim.esm forms that the ESP file modifies, and are thus unable to merge such changes.

The fix for this is to temporarily turn off the 'Use Localized Strings' flag in the Skyrim.esm header. You can do this by opening Skyrim.esm in a hex editor, and changing the ninth byte from 81 to 01. First make a backup of Skyrim.esm just in case this goes wrong!

The start of the file looks like this by default:

54 45 53 34      2C 00 00 00     81 00 00 00

Change it to:

54 45 53 34      2C 00 00 00     01 00 00 00

Save the change and load Skyrim.esm in the CK. You'll get a good number of errors about strings and such, ignore them and press Yes to All. Now it's possible to Check Out forms that belong to Skyrim.esm. Merge any changed Skyrim.esm forms from your ESP into your own ESM, then exit the Creation Kit and revert the flag change.

Advanced Commands

In the folder detail window, where you create your formid list, there are a few advanced commands that can be performed on your .esm. It is unclear of how exactly they function and it appears that you may need to have only the .esm you intend to alter loaded. Otherwise, it will default to affect Skyrim.esm. Very few of these have been tested and the only one I can confirm works is CTRL-SHIFT-B. If you are looking to try any of these, be sure to make a backup in a location other than your merging or Skyrim folder.

The following commands are from Rocket's tutorial on TESNexus's wiki: http://wiki.tesnexus.com/index.php/Version_control_primer

Ctrl+Shift+T

New form type string

Ctrl+Shift+U

---------------------------
Confirm Size Update.
---------------------------
You have chosen to update the size of the selected group.
This operation cannot be undone.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+I

---------------------------
Confirm ID Update.
---------------------------
You have chosen to update the form ID for one or more forms from the current file. 
This operation cannot be undone. Additionally, this will not remove any links in 
other forms to the old ID. Errors may be encountered on load.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+O (letter O)

---------------------------
Confirm ESM Conversion.
---------------------------
You have chosen to create a version of the current ESM file optimized for XBox loading.
This operation cannot be undone.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+P

---------------------------
Confirm Child Sort.
---------------------------
You have chosen to sort all child sub groups in the file.
This operation cannot be undone.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+A

---------------------------
Confirm Bit Array File Validation.
---------------------------
You have chosen to validate the bit array files for the current file.
This operation may take a long time to complete.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+S

---------------------------
Confirm Form Sort.
---------------------------
You have chosen to re-sort the forms in the current file.
This operation cannot be undone and may take a long time to complete.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+F

---------------------------
Confirm FORM Size Update.
---------------------------
You have chosen to update the size of the selected form.
This operation cannot be undone.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+G (Don't use this. You'll regret it.)

---------------------------
Confirm Group Update.
---------------------------
You have chosen to update the form groups in the current file.
This operation cannot be undone and may take a long time to complete.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+L

---------------------------
Confirm ESM Conversion.
---------------------------
You have chosen to create a version of the current ESM file with
Tamriel land only. This operation cannot be undone.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+Z

---------------------------
Confirm Version Update.
---------------------------
You have chosen to update the file version of the master file.
This operation cannot be undone.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+C

---------------------------
Confirm Group Update.
---------------------------
You have chosen to update the child group of the selected form.
This operation cannot be undone.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+V

---------------------------
Confirm File Validation.
---------------------------
You have chosen to validate the loaded forms against the current file.
This operation may take a long time to complete.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+B

---------------------------
Confirm Bit Array File Update.
---------------------------
You have chosen to create/update the bit array files for the current file.
This operation cannot be undone and may take a long time to complete.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+0 (number 0)

---------------------------
Confirm Group Removal.
---------------------------
You have chosen to remove any empty groups in the current file.
This operation may take a long time to complete.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Ctrl+Shift+Delete

---------------------------
Confirm Delete.
---------------------------
You have chosen to remove one or more forms from the current file.
Unlike setting the ignore flag, this operation cannot be undone.
Additionally, this will not remove any links in remaining forms to
the deleted ones. Errors may be encountered on load.

Do you wish to continue?

---------------------------
Yes No
---------------------------

Perforce Integration

Perforce is a 3rd Party Version Management software suite, available for free to groups under 20 members. When set up with the settings documented here, Perforce works hand-in-hand with the Version Control system to keep backups of previous iterations of scripts, and allow for each individual working on a project to pull the latest versions of all project scripts from the Perforce server. The full functionality of it has not yet been uncovered and tested, but research indicates that it may also handle version control for the DialogueViews folder.

With Perforce integration unlocked, it is possible for networked teams working off of a master server using just the Creation Kit and Perforce from a common server for version control.

Setup

First, set up Version Control per the above tutorial.

Download the p4com.dll from this page and add it to your Skyrim folder.

http://public.perforce.com/guest/robert_cowham/perforce/API/p4com/main/

In SkyrimEditor.ini in your Skyrim folder, make sure that, under the [Papyrus] Settings, you have the following set as it is below:

sScriptSourceFolder = "Data\Scripts\Source"

sScriptCompiledFolder = "Data\Scripts\"

sCompilerFolder = "Papyrus Compiler\"

bPerforceEnabled = 1

www.gameindustry.about.com/b/2012/07/29/version-control-management-and-games.htm

I worked off of this tutorial by Warren Schultz, with a few changes that are necessary for the Creation Kit to properly read your script backup workspace. I also ignored some of the instructions more geared towards setting up Perforce on a server that would host multiple users. If you are aiming to collaborate over a network, you may want to heed them.

I will outline the steps roughly for now, but for step by step instructions please refer to that tutorial.

From http://www.perforce.com/downloads

Install Perforce Server using the suggested default settings. Make your username the same as your Windows username. You must have administrator permissions.

Set up your Perforce Depot to have the name TESVData. This is important, the depot name and subfolders that we will eventually set up in our workspace are paths that are hardcoded into the Creation Kit.

Next, go back to the Perforce site and install the Perforce Visual Client, also using default settings. You can uncheck the Plug-in for Windows Explorer.

Open up your Perforce Server, ensure it is set up per the tutorial, then open P4V (the visual client). You will need to create a new workspace named Papyrus (not sure if this is hardcoded but it seems like it may be. Feel free to test other workspace names).

The workspace root should be in a new folder anywhere on your computer. I have mine in the folder above my Skyrim directory, since that is where the Source folder needed for LOD generation is kept. Don't add files yet if prompted.

In Windows Explorer, open up the new directory you created and add the following folders:

\current\Source\scripts \current\Source\DialogueViews

Copy all of the contents of your Skyrim/Data/Scripts folder into the Scripts subfolder. This should be thousands of .psc files and the subfolders for each DLC. If you are using a custom prefix for your project, you will need to set it in your Creation Kit preferences as. Do not rename script fragments manually, even through the Creation Kit.

Refresh your Perforce Client and you should see the new subfolders and files. Right click on the scripts folder and click add. It should put your files into a changelist. Then, right click again and click submit to commit those changes to the depot. You will need to add a comment, it can be anything.

You should now be able to load up the Creation Kit and go to your preferences. At the very end is a tab for Perforce integration, where you can select your new workspace. Select it and hit apply. You shouldn't get any error messages but if you do one of the steps earlier went wrong.

Troubleshooting

Errors

MASTERFILE: File index # is invalid. Clamping to ##. Does a master file depend on more than one master?

The above is a serious error resulting from having an active .esp with more than three master .esm files (including Skyrim.esm). It will corrupt both the .esm file you are merging into as well as the .esp file you are trying to merge if ignored. It is possible to bypass this error, allowing for one or more additional master files as long as all master files from position 02 or lower have no overlapping forms. The best way to do this is to manually renumber the forms in your fourth master file to a higher formid range, such as XX600000+ or XX7000000+ in TES5Edit prior to development.

The following error advises that Perforce integration is not set up. It indicates that the Creation Kit will fall back on default code to assign new formids to script fragments, deleting the original versions without creating backups. It can be safely ignored, but be sure to keep manual back-ups.:

Unable to initialize PerForce, check your logs for more information.

The below error indicates that the Creation Kit has recognized your Perforce workspace, but the directory path is not correct. Your depot must be named TESVData and your scripts folder must be in //current/Source/Scripts where // is the root of your Perforce workspace.

Perforce workspace does not contain a script folder, Papyrus scripts will not be checked in or out.

Notes

In order to merge Navmesh forms (NAVM), the NavMeshInfoMap (NAVI) in Skyrim.esm must also be merged.