UHD Feature Request - verify M2TS hashes

Discussion in 'AnyDVD HD (Blu-ray issues)' started by llowrey, Mar 26, 2018.

  1. llowrey

    llowrey Member

    AnyDVD will silently pass bad sector reads from UHDs. I often have to rip multiple times (iso) to get two rips that are identical.

    The hashes (stored in AACS/ContentHash###.tbl) only apply to the encrypted data. I'm currently ripping protected ISOs so I can run a hash verifier to make sure my iso is good. I then mount the protected iso in VCD and let AnyDVD do the decryption.

    It would save me a lot of hassle if AnyDVD would verify the M2TS hashes and retry until the data is good.

    IMO the fact that AnyDVD returns data that may or may not be correct is a huge problem. AFAIK, this issue only affects UHDs so the UHD support caveat applies but this is a significant enough of a problem to warrant a prominent disclaimer.

    The hash checks can only confirm that the M2TS files are correct (except the last size mod 196,608 bytes) so a correct ISO is not guaranteed but that's the data that matters most. Further checks could be made to ensure that the files in AACS/DUPLICATE and BDMV/BACKUP match their respective entries in AACS and BDMV.
     
  2. thetoad

    thetoad Well-Known Member

    I have makemkv report hash failures on a regular bluray drive that is failing, wouldn't have realize otherwise
     
  3. llowrey

    llowrey Member

    Yes, MakeMKV definitely verifies the hashes, as does DeUHD. If you go back and look at the history of DeUHD it used to be SOP to rip multiple times to get a consistent rip then they added hash checking and folks reported no longer needing to re-rip. AnyDVD is late to the party.
     
  4. Ch3vr0n

    Ch3vr0n Translator NL & Mod

    Feel free to use them then if you like/depend on that function so much. AnyDVD is AnyDVD, not AnyUHD or AnyMKV.

    Sent from my Nexus 6P with Tapatalk
     
    BRCS and whatever_gong82 like this.
  5. llowrey

    llowrey Member

    FTFY

    I would like to see AnyDVD use these hashes to ensure that the data it returns is correct.

    The trouble is that the hashes are of the encrypted M2TS contents. So, if one is using AnyDVD to decrypt the UHD there's no way from the user's end to verify that the contents are correct. We expect a read error to occur when there is a read error, not for random data to be returned with no indication of whether the data is correct or not. AnyDVD could report those read errors by checking the hashes or avoid them by retrying more aggressively until the hashes match.

    Without these checks, folks are going to rip their UHDs, then sometime later (days, weeks, months, years) play them back and possibly encounter video or audio glitches (or worse) due to a bad AnyDVD rip.
     
    Last edited by a moderator: Apr 4, 2018
  6. Ch3vr0n

    Ch3vr0n Translator NL & Mod

    If you're going to quote me, that's fine, but do it right. Don't change my words into something I didn't say. Your post has been edited to revert that change.

    And for the record, I'm not saying it's a bad idea, all I'm saying is if you want/need that function so much use those third party softwares then. It's up to James to decide if/when he wants to add such a function.

    Sent from my Nexus 6P with Tapatalk
     
    Journbon, whatever_gong82 and BRCS like this.
  7. BRCS

    BRCS Well-Known Member

    Most of the read errors of UHD discs are caused by people that do not clean the UHD movies with a microfiber cloth before ripping them they are not Blurays and it is a must to make sure you do this before ripping a new UHD or used movie common sense and you will not have any read errors given you don't have a bad uhd disc or your drive is dirty.
     
    SamuriHL likes this.
  8. ryley

    ryley Member

    I think llowrey's concern is not with the frailty of UHD discs, but instead with AnyDVD not verifying the data it reads from the disc to the greatest extent possible. I would much rather be informed during the ripping process that the disc is dirty/damaged.
     
    John-John and gattopollo like this.
  9. BRCS

    BRCS Well-Known Member

    That is understandable I was just pointing out that UHD discs are very finicky more so than even the dvd format I have ripped over 70 UHD movies and anytime there was an error ir was because i forgot to clean the disc before decrypting it.
     
  10. ryley

    ryley Member

    Are you concerned about your rips suffering from corruption that was not detected by AnyDVD during the ripping process? Because it seems that is a possibility, in addition to typical read errors that cause warnings/errors.
     
  11. BRCS

    BRCS Well-Known Member

    short answer NO I clean my drives and clean the UHD every time before ripping them I learned after the first couple I did that it is a must just trying to pass along the info.
     
  12. llowrey

    llowrey Member

    Example: Factory fresh Sony triple layer disc (Close Encounters of the Third Kind), brand new drive (only a handful of rips under its belt), four iso rips, no errors reported, three different SHA-512 hashes.

    If you get read errors, you have read errors, if you don't get read errors you may or may not have read errors. The best way to know for sure is to rip to protected iso then use another tool to verify the contents using the hashes in the AACS dir.

    Incidentally, I have found that at least sometimes, the drive will return a sector containing MPEG TS packets with the Transport Error Indicator (TEI) flag set to true, indicating that the packet contains uncorrectable errors. Players should handle those and recover appropriately. It appears that the drive knows the sector is bad and returns something the stream processor (player) can deal with.

    The glitches from these bad reads may be fairly minor. I started down this rabbit hole after encountering some major glitches.

    Anyway, all I'm saying is that if you want to make sure the rip is correct, complete, and valid, then more checks are necessary. The fact that no errors are reported is insufficient.
     
    gattopollo likes this.
  13. Steve55

    Steve55 Well-Known Member

    This is so obviously a very desirable feature, I don‘t understand why people are getting het up about it.
    UHD is like music cd‘s it has provision for error correction. So you can get a suboptimal rip and not know.
     
  14. James

    James Redfox Development Team Staff Member

    If you get read errors (error correction of drive fails), AnyDVD should report them. If it doesn't, it is a bug which needs to be fixed.
    If drives don't report read errors but deliver bogus data instead, this would be ... fatal.

    I doubt, that a drive can do that on the fly, with AACS encryption present.
     
    nebostrangla and el Filou like this.
  15. llowrey

    llowrey Member

    It's not uncommon to rip the same UHD multiple times with no errors reported and get isos that are different (according to sha-512). Since no error is reported my suspicion is that the drive is not reporting the error. I doubt that the drive is reporting the error in a way that AnyDVD is just not noticing.

    This was a problem early on with DeUHD. Folks used to routinely rip multiple times until they got a consistent rip. DeUHD then implemented the hash checking and then folks reported no longed needing to rip multiple times. MakeMKV checks the hashes too. I would be surprised if all three (DeUHD, MakeMKV, and AnyDVD) had the same read error bug. I would also be surprised if DeUHD and MakeMKV implemented the hash checks just for fun.

    IMO, hash checking is a necessary feature due to the behavior of the LG firmware of these "friendly" drives. We're never going to get a fix that preserves the "friendliness" and I doubt any other drives will be "friendly" so we're stuck with this behavior.

    For me it's been rare for factory fresh 2-layer discs but maybe 50/50 for new 3-layer discs. Discs with some wear clean up well and maybe show this behavior 20% of the time when cleaned.

    I have never observed this when ripping anything other than UHDs. I believe this to be a characteristic of the "friendly" LG drives, or more specifically the firmware.

    When I saw the TEI flag set I re-ripped the iso unti I got a valid rip and the TEI flag was not set on that iso. I had assumed the drive had set the flag on purpose but I suppose it could have been a random bit-flip due to an bad read.
     
  16. James

    James Redfox Development Team Staff Member

    AnyDVD's decrypting is done in the filter driver (DeUHD & PassKey copied this approach), so a lot of funny things can happen on "the way up" to the application.
    Did you ever get different results, when ripping protected images?

    A "bit flip" on encrypted content will destroy whole blocks. Weird.
     
    nebostrangla likes this.
  17. Steve55

    Steve55 Well-Known Member

    Can't claim to have exhaustively tested, but I did double check exactly that point, and not so far.

    AIUI this is by design. Though all my knowledge comes from other forums. I too am using LG.
     
  18. James

    James Redfox Development Team Staff Member

    I suspect so.
    What is by design?
     
  19. James

    James Redfox Development Team Staff Member

    My suggestion: Use CloneBD directly from the original disc. It will re-read, if it encounters incorrect data (which it will on unnoticed read errors), until "all is well".
     
    BRCS and nebostrangla like this.
  20. thetoad

    thetoad Well-Known Member

    I think the point is that some drives don't return errors, just incorrect data so programs that don't use the hashes think all is fine when it's really not. I'm surprised this happens (I'd figure data should be crc or whatever Blu-rays use and be an error if it falls), but its happened to me so i see the value.

    Sent from my SM-G955U using Tapatalk
     
    Merkapto likes this.