69 Commits

Author SHA1 Message Date
Timmy Welch
22d59aa221 Move HashStorage to its own package 2025-05-31 19:00:40 -07:00
Timmy Welch
ed0b5ba441 Don't include sqlite/vptree in gokrazy build 2025-05-31 18:35:18 -07:00
Timmy Welch
d240bd953b Remove references to loaded json object 2025-05-31 17:59:19 -07:00
Timmy Welch
bb64d8449c Improve memory efficiency 2025-05-31 15:00:05 -07:00
Timmy Welch
b0a4c2939c Map to slice 2025-05-27 22:31:11 -07:00
Timmy Welch
3e364b1858 Filter out non-json files 2025-04-24 12:54:40 -07:00
Timmy Welch
0da76f7fb5 Fix inserting downloaded pages 2025-04-24 12:53:55 -07:00
Timmy Welch
d7c42f5c1d Fix issues with loading saved hashes 2025-04-24 12:46:04 -07:00
Timmy Welch
374c46bc48 Fix url handling order 2025-04-24 12:46:04 -07:00
Timmy Welch
aca658e32d Typo 2025-04-24 12:46:04 -07:00
Timmy Welch
acd71df302 Add the issue number to the downloaded CV pages
Update existing pages 10 at a time if they are missing data
Simplify checking for previously downloaded pages
2025-04-24 12:46:04 -07:00
Timmy Welch
0fd431f6f7 Update result type 2025-04-19 15:41:04 -07:00
Timmy Welch
9322f754bf Fix http server 2025-04-13 16:42:49 -07:00
Timmy Welch
486cf074d0 Remove replace directive 2025-04-13 16:07:49 -07:00
Timmy Welch
bf23bb5a4c Add Server http Header and add -listen and -debug options 2025-04-13 15:58:34 -07:00
Timmy Welch
d7946c2aaf Fix simple results 2025-04-13 15:57:29 -07:00
Timmy Welch
f54b4b1d9d Fix exact matches to match previous behavior 2025-04-13 15:54:17 -07:00
Timmy Welch
a2765b0582 Re-implement sqlite 2025-02-23 12:31:23 -08:00
Timmy Welch
1f37684862 Fix loading saved hashes 2025-02-23 12:31:09 -08:00
Timmy Welch
5307b5d8df Update pre-commit 2025-02-23 12:29:23 -08:00
Timmy Welch
130b7dec4a Update internal hash storage 2025-02-22 13:45:41 -08:00
Timmy Welch
6452f2e50d Update hashes from comicvine to 02-06-2025 2025-02-06 16:19:05 -08:00
Timmy Welch
f52219cb31 Fix loading old hashes 2025-02-05 17:59:59 -08:00
Timmy Welch
5a93dacdad Fix checking for new covers 2025-02-05 16:32:25 -08:00
Timmy Welch
798ae49c8b Update dependencies 2025-01-29 16:53:47 -08:00
Timmy Welch
8d6db630d4 Fix channel initialization 2025-01-21 18:26:12 -08:00
Timmy Welch
29f58e7fe7 Limit download buffer pool to 10MB buffers 2025-01-11 16:15:36 -08:00
Timmy Welch
fe3f045c6e Fix issues with reading cv files 2025-01-11 15:26:35 -08:00
Timmy Welch
8ce1ca3354 Fix invalid index 2025-01-11 12:19:14 -08:00
Timmy Welch
75d60339ee Optimizations
Re-arrange some channels
set all channels to a size of 1
Reset the image download buffer
Allocate the correct slice size for the CV API
Add a flag to only download the original image
2025-01-10 22:28:51 -08:00
Timmy Welch
8a9aec4884 Don't catch sigabrt 2025-01-10 14:24:12 -08:00
Timmy Welch
033c68593b Implement a buffer pool
Downloads in quick succession could cause an OOM when the GC couldn't
  keep up
2025-01-09 02:07:36 -08:00
Timmy Welch
260a13688a Wait every 200 downloads so CV doesn't get overloaded 2024-12-26 17:50:25 -08:00
Timmy Welch
e04469938d Fix downloading images that are not hashed 2024-12-26 16:11:10 -08:00
Timmy Welch
8bdfb282b9 Only create directories if we are keeping images 2024-12-25 15:22:37 -08:00
Timmy Welch
9baa4dbc17 Add RmdirP 2024-12-25 15:11:11 -08:00
Timmy Welch
5cf25089da Improve downloading images 2024-12-25 14:53:56 -08:00
Timmy Welch
cf2c61b92e Update pre-commit 2024-12-25 14:49:22 -08:00
Timmy Welch
cc4e973bf9 Make deleting files efficient 2024-12-15 14:15:33 -08:00
Timmy Welch
7ede0dee72 Fix file closing 2024-10-31 21:12:53 -05:00
Timmy Welch
840e97cff0 Handle reading existing invalid json 2024-10-19 09:50:23 -07:00
Timmy Welch
50fcfb9513 Remove downloaded result on json failure 2024-10-19 09:41:42 -07:00
Timmy Welch
776ca68e3e Wait for page download failures 2024-10-19 09:39:08 -07:00
Timmy Welch
95fa44fb97 Fix mutex on badURLs 2024-10-18 14:35:29 -07:00
Timmy Welch
d0c7ed792a Fix infinitely increasing url length 2024-10-18 14:30:30 -07:00
Timmy Welch
97802d9111 Start cover download immediately 2024-10-18 11:48:06 -07:00
Timmy Welch
89c9b4ebce Allow truncating image files to save space 2024-10-18 11:19:50 -07:00
Timmy Welch
a65cadf106 Add CV cover downloader 2024-10-18 10:50:35 -07:00
Timmy Welch
4922ceb678 Fix locking for map storage 2024-10-16 17:56:19 -07:00
Timmy Welch
da54b3a454 Fix decoding hashes 2024-10-14 02:02:26 -07:00