Timmy Welch
75d60339ee
Optimizations
...
Re-arrange some channels
set all channels to a size of 1
Reset the image download buffer
Allocate the correct slice size for the CV API
Add a flag to only download the original image
2025-01-10 22:28:51 -08:00
Timmy Welch
033c68593b
Implement a buffer pool
...
Downloads in quick succession could cause an OOM when the GC couldn't
keep up
2025-01-09 02:07:36 -08:00
Timmy Welch
260a13688a
Wait every 200 downloads so CV doesn't get overloaded
2024-12-26 17:50:25 -08:00
Timmy Welch
8bdfb282b9
Only create directories if we are keeping images
2024-12-25 15:22:37 -08:00
Timmy Welch
5cf25089da
Improve downloading images
2024-12-25 14:53:56 -08:00
Timmy Welch
cc4e973bf9
Make deleting files efficient
2024-12-15 14:15:33 -08:00
Timmy Welch
7ede0dee72
Fix file closing
2024-10-31 21:12:53 -05:00
Timmy Welch
840e97cff0
Handle reading existing invalid json
2024-10-19 09:50:23 -07:00
Timmy Welch
50fcfb9513
Remove downloaded result on json failure
2024-10-19 09:41:42 -07:00
Timmy Welch
776ca68e3e
Wait for page download failures
2024-10-19 09:39:08 -07:00
Timmy Welch
95fa44fb97
Fix mutex on badURLs
2024-10-18 14:35:29 -07:00
Timmy Welch
d0c7ed792a
Fix infinitely increasing url length
2024-10-18 14:30:30 -07:00
Timmy Welch
a65cadf106
Add CV cover downloader
2024-10-18 10:50:35 -07:00
