[ Resolved] 1.2.5.0 Batches and Quotas (app freezes)

[Kerry]

Howdy, and Happy Thanksgiving to all from Texas, USA!

First let me thank the devs for a wonderful app, and great features. I've been using AS for a few years now and I have loved each new update, but the latest major feature add with batch downloads is awesome! This is a fantastic addition and I love it! Been using it since Beta and have had 1.2.5.0 since it came out. I haven't tried 1.2.6.x yet and I think I'll wait a bit (seeing issues reported with NF logins - I can wait a while, 1.2.5.0 seems fine to me for now anyway).

I have noticed a behavior with Batch downloads related to Quotas that can cause the program to freeze every time (no crash, no dump, no logs) - and I'm not complaining at all, just want you to be aware (I know how to avoid it, so I do)...

If I have two batches queued up in different services, such as AP and HMx, and the quota bucket empties out, and both batch queues are waiting for the next available token drop to splash in the empty token quota bucket, both service queues try to grab the next available token at the exact same moment as soon as it becomes available, and the app freezes. Have to kill the app with Windows task manager and restart it.

No big deal, I just set up batches for one service at a time and all is well, but it would be nice at some point to find a way to have multiple batch queues set up in multiple services and not worry about an overnight app freeze.

Which kinda brings me to another point - since the token quotas are meant to help avoid issues with providers, couldn't each provider have their own independent quota bucket? (Sorry if this has been mentioned before). AP shouldn't care about traffic from NF or D+ for instance. Oh well, just a thought.

Thanks again for helping me fill my Plex lib with 3,000+ movies and 100+ TV shows with >25k episodes so far. (AnyDVD has been a long time great help there too - keep up the great work!)

 

[DeepSpace]

If I have two batches queued up in different services, such as AP and HMx, and the quota bucket empties out, and both batch queues are waiting for the next available token drop to splash in the empty token quota bucket, both service queues try to grab the next available token at the exact same moment as soon as it becomes available, and the app freezes. Have to kill the app with Windows task manager and restart it.

You can set the amount of simultaneous videos to download in settings. I haven't tried multiple providers yet so this had no affect for me yet, but I'm using batch for quite a while and had no errors, even with adding more than one season.

couldn't each provider have their own independent quota bucket?

Yes this has been mentioned countless times, but atm it will stay at is. If it should change, you should see it in the changelog.

 

[Kerry]

You can set the amount of simultaneous videos to download in settings. I haven't tried multiple providers yet so this had no affect for me yet, but I'm using batch for quite a while and had no errors, even with adding more than one season.

Thanks, I haven't looked at the simultaneous videos to download setting yet. I'll take a look. Haven't had a problem at all with multiple seasons within a single series / provider (I've acquired over 500 Looney Tunes episodes ok from multiple seasons - they all queue up in a single batch from one provider). The issue only occurs when multiple providers have separate batches queued and more than one provider queue needs the next available token at the same time. But like I said that is easy to avoid by just batching stuff for one provider, let it finish, then batch another provider.

 

[Pete]

If I have two batches queued up in different services, such as AP and HMx, and the quota bucket empties out, and both batch queues are waiting for the next available token drop to splash in the empty token quota bucket, both service queues try to grab the next available token at the exact same moment as soon as it becomes available, and the app freezes. Have to kill the app with Windows task manager and restart it.

That is quite strange, because this exact scenario has been tested over and over.
The queue knows exactly how many tokens are available and normally should not start two providers simultaneously. I'll do some testing...

 

[Pete]

@Kerry I cannot reproduce this.
With zero tokens available and two or more providers waiting, only one gets the OK and the others continue waiting. And the program logic is pretty clear there as far as I can see.

If you ever manage to reproduce this (that would be VERY helpful), please use the task manager to create a crash dump, so I can analyze the situation. It really sounds just like a thread-deadlock and I'd need to identify the cause.

If you contact me via PM with your license number and we can agree on a time, I can artificially "empty" your bucket for testing and when the result is there, I'll fill it up again.

 

[Kerry]

That is quite strange, because this exact scenario has been tested over and over.
The queue knows exactly how many tokens are available and normally should not start two providers simultaneously. I'll do some testing...

Thanks - I had a series queued in HMx and another in AP, if that helps in your testing. The HMx series was Looney Tunes, which burns through tokens fast as each episode is only about 6 minutes, so it doesn't take long to chew through the quota when getting shorts like that. Constant state of waiting for the next one to drop in the bucket. Appreciate your help.

 

[Kerry]

@Kerry I cannot reproduce this.
With zero tokens available and two or more providers waiting, only one gets the OK and the others continue waiting. And the program logic is pretty clear there as far as I can see.
If you ever manage to reproduce this (that would be VERY helpful), please use the task manager to create a crash dump, so I can analyze the situation. It really sounds just like a thread-deadlock and I'd need to identify the cause.
If you contact me via PM with your license number and we can agree on a time, I can artificially "empty" your bucket for testing and when the result is there, I'll fill it up again.

Sure - I'll try to set up a similar test this weekend and will get a task manager crash dump this time. Would knowing my OS and hardware help? (Win 10 64 v21H1 - Dell OptiPlex 3050 - Processor Intel(R) Core(TM) i5-7500T CPU @ 2.70GHz - Installed RAM 8.00 GB (7.73 GB usable)

 

[Kerry]

Oh - currently my AS+ queue download setting is at 3 concurrent, and I am downloading everything at 1x speed (Realtime)

(And everyone please refrain from mocking my little rig - all I use this one for is AS+ and Plex media server - it may be small but it's got 24TB storage! )

 

[Kerry]

@Pete ... I have a batch series queued in D+ (Simpsons - again) and another batch queued in HMx (Looney Tunes Cartoons season 3)... D+ is currently dl an episode on the most recent token now (I added the HMx batch during this dl) - so when this current dl completes I will be at 0 quota again, and both providers will be waiting for the next token - in theory this should trigger what happened before and it will freeze again when that next free token drops in the quota bucket ... I'll report what happens and will get a crash dump if it freezes again as before.

 

[Kerry]

@Pete the app did not freeze this time - HMx and D+ both fought over the same next free token when it arrived, HMx won and D+ tried a DL then aborted (see screen shots below) - since D+ tried on a failed token that it lost, it created an empty dl file, which means the next dl attempt will see that file and will ask if it should be overwritten or not (which I assume will mean the app will be waiting for a user response to the dialog request and that batch queue will be frozen until a human appears and responds - not sure if the HMx queue will keep chugging along ok in its own thread or not)... I included the log from the D+ attempt...

Two queues Waiting for next token...


Next token arrived:

 

[X3rx3s]

if it helps, ive experienced this same issue....only when i have multiple providers qued up and the token bucket is empty.....i was hoping it was going to go away with 1.2.6.0 but that one had naming issues and i had to revert to 1.2.5.1
if i happen to experience it again ill send log files, but ive also learned how to avoid it....was going to try the concurrent downloads set to 1 if it happens again which should help but doesnt really solve the problem

 

[Kerry]

@Pete Concurrent queues tried to grab the same token again - see two screenshots again below... HMx won the fight again and got the token - D+ lost the token and aborted again - skipped the previously failed / aborted attempt - so basically this behavior will repeat and will burn through the D+ queue, not getting any of them at all...

Waiting for next Token again:



HMx wins the battle for the token - D+ cruises past failed / aborted episode and proceeds down the queue and fails / aborts the next episode:

 

[Kerry]

@Pete - and now it just crashed... two logs / dumps attached, the .zip contains a .dmp file...

 

[Kerry]

@Pete the Windows task manager crash dump DMP file is too large to upload (even zipped, it is 571Mb) ...could not close the app any other way except via Task manager - even the crash report dialogs were not responding.

 

[RedFox 1]

This only happens when you overload the queue. You need to use 2 or 3 providers and load it up with at least 30 to 40 episodes, if 2 ends at the same time you get a deadlock. Its an operating system thing where the screen goes white and you get the loading circle.

 

[Pete]

OK, thanks for all the info, this will not get fixed fast, I'm afraid.

This only happens when you overload the queue. You need to use 2 or 3 providers and load it up with at least 30 to 40 episodes, if 2 ends at the same time you get a deadlock. Its an operating system thing where the screen goes white and you get the loading circle.

I'll try to reproduce it that way somehow. Thanks.

 

[Kerry]

OK, thanks for all the info, this will not get fixed fast, I'm afraid.

:thumbs-up: Thanks @Pete I appreciate the help - no rush, I know how to avoid this issue. let me know if you need me to do any more test crashes and send more logs / dumps.

 

[Kerry]

This only happens when you overload the queue. You need to use 2 or 3 providers and load it up with at least 30 to 40 episodes, if 2 ends at the same time you get a deadlock. Its an operating system thing where the screen goes white and you get the loading circle.

Thanks @RedFox 1 - appreciate the help. If 2 dl's ended at the same time in my testing here, I think it was due to both queues failing with 0 byte dl's at the same time (because the D+ queue was NEVER able to grab a token while the HMx queue was running, HMx always won the race for the next available token.) Both queues had dl errors at the same time I guess.

If you stack up a couple provider queue with lots of small episodes (like about 20 or more 25 minute sitcoms or 5-6-7 minute shorts from Looney Tunes or Tom & Jerry) you should be able to replicate this issue pretty quick.

 

[RedFox 1]

Thanks @RedFox 1 - appreciate the help. If 2 dl's ended at the same time in my testing here, I think it was due to both queues failing with 0 byte dl's at the same time (because the D+ queue was NEVER able to grab a token while the HMx queue was running, HMx always won the race for the next available token.) Both queues had dl errors at the same time I guess.

If you stack up a couple of providers in the queue with lots of small episodes (like about 20 or more 25 minute sitcoms or 5-6-7 minute shorts from Looney Tunes or Tom & Jerry) you should be able to replicate this issue pretty quick.

A deadlock is a situation in which two downloads sharing the same resource are effectively preventing each other from accessing the resource, which in this case is the token. Pete will fix it but because it also affects the OS and Anystream it will not be easy to resolve, as Pete said. You can easily avoid this by not overloading the queue, the chances are reduced but
not eliminated. I have only had this happen once. The program crashes, you lose all the not downloaded items in the queue and you have to start over. all that said I am not the developer, this is only my assumption. Pete will fix it, that I am sure of.

 

[Pete]

Ok, I understand the problem now, I was able to reproduce it in one out of three tests.
It's a bit complex and depends on timing (in a nutshell: the queue "gives back" a reserved token just a tiny bit too soon and then for a brief moment, there appear to be more tokens available, than there really are). The new release - coming up - should fix this.
Thanks for your excellent help, you too, @RedFox 1