UHD Feature Request - verify M2TS hashes

[James]

I think the point is that some drives don't return errors, just incorrect data so programs that don't use the hashes think all is fine when it's really not. I'm surprised this happens (I'd figure data should be crc or whatever Blu-rays use and be an error if it falls), but its happened to me so i see the value.

CloneBD knows, how correct data should look like and will automatically retry, if the data doesn't look "good".

 

[SamuriHL]

And honestly, I'd question owning a drive that did that. Just a personal opinion, of course.

 

[thetoad]

And honestly, I'd question owning a drive that did that. Just a personal opinion, of course.

It worked for a long time fine )I've had it for 5 plus years, the only reason i knew it was going bad was because makemkv threw those hash errors)

Sent from my SM-G955U using Tapatalk

 

[jwjohnson]

My suggestion: Use CloneBD directly from the original disc. It will re-read, if it encounters incorrect data (which it will on unnoticed read errors), until "all is well".

I just tried creating a full disc ISO from The Last Jedi UHD (Best Buy Steelbook) using Clone BD. I had problems previously with errors when creating an MP4 from the ISO I made using AnyDVD (no errors reported by AnyDVD). The process of creating the ISO starts out great, but when the sketchy area of the disc was encountered the read speed dropped to 1000-1500 kbps and instead of the initially projected completion time of 1 hour 23 minute it is now projecting 18 hours and 53 minutes. I understand slowing down to re-read the problem area until the data is consistent, but seems like it would make sense to go back to the original read speed once good data is again encountered.

Update: It ended up taking almost 4 hours to complete the ISO. I'm now going to try creating an MP4 from it with Clone BD to see if the data is intact. I've attached a Clone BD log of the ISO creation from the original UHD.

 

[llowrey]

Did you ever get different results, when ripping protected images?

Yes. I haven't been ripping protected isos until just recently so I didn't have much data. I re-ripped a bunch of discs and found that protected iso don't silently fail as often but it does happen.

For example, I ripped X-Men First Class to protected iso, got zero errors reported, yet when I check the hashes, 2,652 of 126,983 hash blocks of layer 1 were mismatched, but (interestingly) 0 of 146,990 of layer 2 were mismatched. I'll re-rip several more times to see how the data varies, in case anyone is interested.

Edit1:

Hash mismatches

Rip 1: 2,652 of 126,983 of layer 1 and 0 of 146,990 of layer 2
Rip 2:   465 of 126,983 of layer 1 and 0 of 146,990 of layer 2
Rip 3: 2,998 of 126,983 of layer 1 and 0 of 146,990 of layer 2
Rip 4: 2,652 of 126,983 of layer 1 and 0 of 146,990 of layer 2

SHA-512's of the protected ISOs

Rip 1: 1d7be5e095e6a58386ca97bcb493b52ec3b88248f5df3dfc7398345d8bb82f08b78a7467ed6c72ebecf4558de62b127804ccab9e9ac89c72084d244d7dcd4830
Rip 2: ca50b852c4ba8b06fbe71ec00adbfd06c67d29386256a56c6854c8b32cd87a0044d47f58f27af1da0d6c6a13290352c770fb7aae6bae61830f74dc1b5a9c88db
Rip 3: 7debd3db6f8620bdb55be5e40f8f1fc29a9c32cc0829c17458933e548ecbb0af461e03b2714794eeb71fa9abaae5036f7164c477b554b02c8f06a15b1b629479
Rip 4: e92c16d91c67c7dff59d066ae5adac41cb6f727a56fe5ff67df93a32b0ab9969228f8751468082f8f4b728ddf697e8f59d5d6001af3b705a57bd72d1fc776d2f

 

[jmone]

I don't want to go back to the early days (DeUHD) of double ripping to check the rips are "good". I just check a double rip using AnyDVD HD (one directly and one from JRiver Front end with AnyDVD running the the background) and the main M2TS has different hashes (rest of the files matched). Same disc, same drive just an hour apart. AnyDVD is my prefered ripper but... I'm not sure what to do, as while I have licences for all 3:
1) AnyDVD does not verify rips (is this true?)
2) DeUHD has issues with its drivers in the latest version of Windows
3) MakeMKV's m2ts are different to AnyDVD/DeUHD (different flag settings)

 

[llowrey]

I am able to verify about ~99.5% of protected ISOs by checking the following:

1. Everything in /AACS matches what's in /AACS/DUPLICATE
2. Everything in /BDMV, /BDMV/CLIPINF, and /BDMV/PLAYLIST matches what's in /BDMV/BACKUP, /BDMV/BACKUP/CLIPINF, and /BDMV/BACKUP/PLAYLIST
3. All hash units defined in /AACS/ContentHashNNN.tbl match the M2TS files
4. All JARs are valid (no CRC errors)

Not all dirs in /BDMV are required to have a backup (just CLIPINF and PLAYLIST are) but some discs do include other dirs in /BDMV/BACKUP so a few more checks are possible.

The UDF metadata is duplicated on the disc so that can be checked.

What remains are files not included in /AACS/DUPLICATE and /BDMV/BACKUP, unallocated space, and, most importantly, the tails of the M2TS files.

A hash unit is a block of 96 sectors (actually defined as 3 ECC units where an ECC unit is 64 sectors). That works out to 192K (196,608). Only M2TS files have hashes. Any M2TS smaller than a complete hash unit (96 sectors) is excluded. The remainder of any M2TS which is not an integer multiple of 96 sectors in length, is excluded.

The bummer is that the last (1-192K) bytes of each M2TS can't be verified since no hash will exist. If the main movie is a single M2TS then it's not a big deal (to me). However, if the film employs seamless branching, then these unverifiable tails could occur in the middle of the film. So for that I'm relying on multiple ISO rips and taking the majority value for those file remainders. Unfortunately, even though only a small amount of data needs to be re-read, the only way to get the raw content is to rip a full protected ISO which takes more than an hour.

 

[Rupan]

Note that the content hash table itself is hashed, with checksums available in the content certificate. There really isn't a need to compare to backups.

The AACS-LA switched to truncated SHA-256 in the content certificates on UHD discs but the spec appears otherwise similar to the BD pre-recorded book spec from AACS 1.0.

 

[Pete]

but when the sketchy area of the disc was encountered the read speed dropped to 1000-1500 kbps and instead of the initially projected completion time of 1 hour 23 minute it is now projecting 18 hours and 53 minutes. I understand slowing down to re-read the problem area until the data is consistent, but seems like it would make sense to go back to the original read speed once good data is again encountered.

That, you'll have to discuss with the drive manufacturers.
Slowing down the drive on read errors is not a software thing. The drive reduces read speed, when it encounters read errors and spins back up after an undefined "while".

 

[llowrey]

Note that the content hash table itself is hashed, with checksums available in the content certificate. There really isn't a need to compare to backups.

The AACS-LA switched to truncated SHA-256 in the content certificates on UHD discs but the spec appears otherwise similar to the BD pre-recorded book spec from AACS 1.0.

Yes, however, there are files other than ContentNNN.cer and ContentHashNNN.tbl that are important and if you are verifying those others against their AACS/DUPLICATE counterparts you might as well verify them all. Plus, all of the BDMV/BACKUPS need to be checked.

The ContentNNN.cer fiels are also cryptographically signed by a certificate so their contents can be verified. I have yet to figure out what that cert is. Do you know? The hashing of the M2TS files is for detecting tampering, not for verifying data integrity, so I doubt the cert is on the disc.

 

[Rupan]

The content certificates for each layer contain "Signature data" at the end, but it appears that the signature size and algorithm are not specified anywhere in the AACS BD pre-recorded content book. Unfortunately I don't have the public certificate nor do I know where it might be found. My guess is that this is the AACS-LA root certificate, but it might also be some intermediary certificate signed by the AACS-LA. I would also note that the AACS specification would seem to indicate that, for non-UHD blu-ray, the full SHA-1 hashes are stored in the content hash table (see the note on page 6, where it says "K = 88+(J-1)*20"). However, on the discs I've seen, only the lower 8 bytes of the hash values are actually stored so the documentation is wrong in some cases.

@Pete or @James, would you be willing to provide some insight as to the signature algorithm and signature data sizes?

 

[jmone]

Thanks James for adding the new verification feature. Will this work with AnyDVD running in the background with other apps ripping, or only when AnyDVD rips to folder?

 

[Rupan]

Indeed, thanks for adding this feature in the latest beta.

 

[James]

Thanks James for adding the new verification feature. Will this work with AnyDVD running in the background with other apps ripping, or only when AnyDVD rips to folder?

As it is a feature of the AnyDVD ripper, it will only work with the AnyDVD ripper. Copy to folder or unprotected image will both work.

 

[Rupan]

Why not protected images as well? If the data is damaged on read when creating a protected image, then subsequent decryptions will also be damaged.

 

[James]

Why not protected images as well? If the data is damaged on read when creating a protected image, then subsequent decryptions will also be damaged.

Correct. But the AnyDVD ripper can't analyze the decrypted data, if the data ... isn't decrypted.

 

[Ch3vr0n]

@James Isn't there a way then to compare the encrypted data with the data on the disc? Or can't that be done because the ROM mark needs to be involved (which obviously can't be copied over)?

 

[James]

@James Isn't there a way then to compare the encrypted data with the data on the disc? Or can't that be done because the ROM mark needs to be involved (which obviously can't be copied over)?

ROM mark doesn't matter. Verifying by reading twice is possible, but IMHO lame. You could do this yourself by creating two protected images and comparing the hashes AnyDVD creates, if the "create additional..." checkbox is checked.
I can think of something better (ripper decrypting while ripping just to check integrity? Tricky...), but let's test first, how good or bad the current system works...

 

[Ch3vr0n]

I'm just thinking, either way I pretty much always rip to decrypted iso. And I think in my years of using AnyDVD I've never had a single bad rip.

Sent from my Nexus 6P with Tapatalk

 

[llowrey]

Correct. But the AnyDVD ripper can't analyze the decrypted data, if the data ... isn't decrypted.

Why do you need to analyze the decrypted data? Only the m2ts files are encrypted, yes? The hashes are of the encrypted m2ts files, so the only time you need to inspect the decrypted data would be the "tails" of the m2ts files which aren't covered by the hashes. Or... are you not checking the hashes?