Sony PlayStation Vita Dumping and Preservation proposal for No-Intro

General No-Intro related discussions.
Post Reply
iCEQB
Posts: 8
Joined: 11 Aug 2017 12:48

Sony PlayStation Vita Dumping and Preservation proposal for No-Intro

Post by iCEQB » 31 Mar 2019 19:22

Sony PlayStation Vita Dumping and Preservation proposal for No-Intro



This thread will describe the possibility for the No-Intro community to dump and preserve PlayStation Vita games in a state, which is the closest to the phyiscal copy of the game.
I will provide insight of why it wasn't possible before, how it was made possible, why the current method is (very) bad and tools for a smooth transition.


The Problem:

Physical PlayStation Vita games don't contain 100% the same data across multiple cartridges of the same game, even if the game ID, region, version etc. match.
This is because, during the manufacturing state, Sony alters the image that gets written to the games ROM (or flash memory) for each cartridge.

There are two parts which get altered during this process:
  1. A block of unknown data at the very beginning of the games FAT partition. Currently we don't know what this block contains, it might be a watermark or a signature. It's not important for the preservation what this data is or contains, so we will describe this chunk of data from here on as "unk" for unknown.
  2. The second part that is unique for every cartridge of the same game, is the license file. The license file is probably unique, so Sony can identify a copy over the internet, to see if one cartiridge is played multiple times at the same given time. This is probably a countermeassure for piracy.
Another problem was, that no tool was available to create an image of the PlayStation Vita cartridges, like for example on the 3DS or Switch.
Such a dumper thanfully became available as of September 2017 and is called psvgamesd.

The current method that No-Intro uses to preserve PlayStation Vita games is functional for gamers, but very bad in terms of preservation. The VPK method skips "unnecessary" data, it patches the games executables and packs everything in a ZIP like container that does not represent the physical game cartiridge in any way.
It also introduces bugs to the games while patching them - for example in the game selection screen of the "Metal Gear Solid HD Collection" you have no background sound (themes for each game), where it's perfectly functional in the physical and digital (PSN) copy.


The Goal:

The goal with the method written below, is to maintain No-Intros standard, which is to preserve physical games in a digital form, that is as close as possible to the real cartridge as it can be.
As we learned above, this can be a challange if every cartridge contains unique data like on the 3DS, or if (software) dumpers introduce data into the dump, that is not present on the physical copy, like the iNES header.

I will describe a method that strips all data introduced by psvgamesd and null all unique bytes within a PlayStation Vita dump. This method makes sure that No-Intro only preserves all common bytes across multiple cartridges of the same game, while at the same time staying as close as possible to the original media as it can get.

The "problem" one might see here is, that by modifying the dumps as described, will render them unplayable. To the best of my knowledge, it was never the intention of No-Intro to only preserve games in a playable state.

There is currently a method in the works, that will reconstruct these dumps and make them playable again, but for the sake of preservation, this is the only option that allows us to verify a dump with another cartridge from the same game to make sure it's 100% correct.


The Solution:

In the past I already did some research on the PSV images that psvgamesd creates, to identify what data to keep and what to delete from the dumps.
Here you can read about my findings back then, which unfortunately never got any feedback from anyone, let's hope this solution finds more resonance :P.

The steps described here can either be done by hand, or you can test a tool called "PSVStrip", which has been written for this purpose by KippyKip for us.

Step 1:
Install psvgamesd on your hacked PlayStation Vita and use it to dump your cartridges. Copy the resulting PSV file(s) to your computer to a place where you can work with them.

Step 2:
Open a HEX Editor you are comfortable working with (like HxD etc.) and open the PSV file you want to modifiy.

Step 3:
Mark the first 0x200 / 512 bytes and remove them.
This is the PSV header, which gets introduced into the dumps by psvgamesd and is not present on the physical cartridge, so we are not going to preserve this data.

Step 4:
Jump to offset 1C00 and mark 0x260 / 608 bytes and null them - DON'T remove them!
This is the unknown chunk of data that I described in the introduction. I tried to find information of what it could be, I even talked to well known Vita developers in the scene but they could also only speculate what it might be.

Step 5:
Now search for the first 16 bytes of a PlayStation Vita license with the following hex value:

Code: Select all

FF FF 00 01 00 01 04 02 00 00 00 00 00 00 00 00
Not every byte of the license is unique and since we want to preserve every common byte across multiple cartridges of the same game, we are only going to null the unique bytes.

So let's say we found the license at offset 64AE0200 - let's call it license base.
We need to null the license at two locations.

1) Jump to offset "license base + 0x50 / 80 bytes" (Offset 64AE0250), mark 0x10 / 16 bytes and null them - DON'T remove them!

2) Jump to offset "license base + 0xA0 / 160 bytes" (Offset 64AE02A0), mark 0x160 / 352 bytes and null them - DON'T remove them!

To clarify the license part a bit, I made a picture of three different licenses from the same game (Wipeout 2048). The blurred parts are the bytes which are unique and need to be nulled (open in new tab to get a bigger image):

Image

Alternatively you can use PSVStrip by KippyKip attached below, which takes care of all these steps for you.
In its current form PSVStrip can modify single PSV images or even entire folders containing multiple PSV files, in case you already built up a collection.

Please run PSVStrip in a command promt to see the usage instructions.

Image

In the output directory you will notice, that PSVStrip creates multiple smaller files next to the patched PSV image. The .hdr, .unk, .lic1 and .lic2 files should be self explanatory. But next to them is also a .psve file, which contains all the stripped data consolidated in one single file. This file can be used to reconstruct the original PSV image, so it can be played again with psvgamesd.

As of right now, this reconstruction function is not implemented yet, it is also up to the dumper if he wants to share his .psve file. How this is going to be handled is up to the administrators of No-Intro. Redump has a similar approach with PS3, XBOX360, Wii U, etc. games, where the dumper can decide if he wants to share the disc key or not (which is not unique btw like the Vitas license data).

There is also a second method to reconstruct the PSV images by using fake licenses, should No-Intro or the dumper decide to not share this data.

So even if the data we will preserve can not be used to play the games as is, it is not useless !


The Catch:

There are certain PlayStation Vita games, that contain a writeable parition called "grw0:" to store data on it (like save data).
These games can be identified by using VitaShells ability to list all devices / partitions, which would look like this:

Image

Games with such a writeable parition can't currently be dumped by psvgamesd, which is a limitation that only a developer with the needed know how can fix. Several people tried to contact the author of psvgamesd in the past and ask if there are any plans to fix this issue, but there was never any feedback to that as far as I know - maybe the preservation effort here will give psvgamesd some life again :)

Here is a small list of games I could find with a writeable partition:

Code: Select all

Epic Mickey 2
Jak and Daxter Trilogy
Touch My Katamari
Virtua Tennis 4
Ninja Gaiden Sigma 2 (unconfirmed)

Credits and closing words:

Attached below you will find PSVStrip and an example XML (DAT) with the following games:

Code: Select all

Girls & Panzer Senshado, Kiwamemasu! (JPN) (PCSG00339)
Hatsune Miku Project DIVA f (JPN) (PCSG00074)
Metal Gear Solid HD Collection (USA) (PCSE00020)
Wipeout 2048 (EU) (PCSF00007)
If you have any of these games, please use PSVStrip or the manual method from above and check for yourself that this is a legit way to preserve PlayStation Vita games from across multiple cartridges.

Should you find any bugs in PSVStrip or see that your modified copy of any game in the example DAT does not match, please contact me ASAP !

Finally I want to thank the following people for helping me defining the method and writing the tools needed to get the PlayStation Vita preservation in the right direction:

- YifanLu: helping with further understanding the PSV image format
- Motoharu: psvgamesd and helping further understand the PSV image format
- KippyKip: author of PSVStrip
- FakeShemp: author of VitaLicenseEditor, the predecessor of PSVStrip, that unfortunately due to time constraints never got finished
- Everyone at VGPC: bugging me to finally finish this task :whistling:

Download PSVStrip
You do not have the required permissions to view the files attached to this post.

Vigi
Posts: 149
Joined: 18 Jun 2008 10:16

Re: Sony PlayStation Vita Dumping and Preservation proposal for No-Intro

Post by Vigi » 31 Mar 2019 21:21

Some noob questions that I have from reading your post:

- How do you know that psvgamesd dumps are really complete and unmodified (besides the added header)? Did anyone check the source code to see if it is performing the correct operations?

- Why blank out the 'unknown block' if you dont know its purpose? Did you check if this data is the same on 2 cartridges of the same game? If it matches between cartridges then it should probably be preserved.

iCEQB
Posts: 8
Joined: 11 Aug 2017 12:48

Re: Sony PlayStation Vita Dumping and Preservation proposal for No-Intro

Post by iCEQB » 31 Mar 2019 22:00

Vigi wrote:
31 Mar 2019 21:21
Some noob questions that I have from reading your post:

- How do you know that psvgamesd dumps are really complete and unmodified (besides the added header)? Did anyone check the source code to see if it is performing the correct operations?
The readme on the github states so and what you get is a raw FAT partition (+ the added header for obvious reasons) from the cart,
- Why blank out the 'unknown block' if you dont know its purpose? Did you check if this data is the same on 2 cartridges of the same game? If it matches between cartridges then it should probably be preserved.
Of course I did check that, else nothing I wrote would make sense :D
Everything that gets blanked out is unique data - please reread what I said above.

Hiccup
Datter
Posts: 642
Joined: 09 Oct 2015 11:29

Re: Sony PlayStation Vita Dumping and Preservation proposal for No-Intro

Post by Hiccup » 01 Apr 2019 09:24

iCEQB wrote:
31 Mar 2019 22:00
Vigi wrote:
31 Mar 2019 21:21
Some noob questions that I have from reading your post:

- How do you know that psvgamesd dumps are really complete and unmodified (besides the added header)? Did anyone check the source code to see if it is performing the correct operations?
The readme on the github states so and what you get is a raw FAT partition (+ the added header for obvious reasons) from the cart,
But is the FAT partition the only data on the chip?

iCEQB
Posts: 8
Joined: 11 Aug 2017 12:48

Re: Sony PlayStation Vita Dumping and Preservation proposal for No-Intro

Post by iCEQB » 01 Apr 2019 10:23

Hiccup wrote:
01 Apr 2019 09:24

But is the FAT partition the only data on the chip?
As posted above, there are certain games with a second partition (grw0:), which are currently not dumpable anyway.
Apart from that, every other game only has one partition and that is dumped RAW by psvgamesd -so yes.

Post Reply