Commit Graph

43 Commits

Author SHA1 Message Date
29f58e7fe7 Limit download buffer pool to 10MB buffers 2025-01-11 16:15:36 -08:00
fe3f045c6e Fix issues with reading cv files 2025-01-11 15:26:35 -08:00
8ce1ca3354 Fix invalid index 2025-01-11 12:19:14 -08:00
75d60339ee Optimizations
Re-arrange some channels
set all channels to a size of 1
Reset the image download buffer
Allocate the correct slice size for the CV API
Add a flag to only download the original image
2025-01-10 22:28:51 -08:00
8a9aec4884 Don't catch sigabrt 2025-01-10 14:24:12 -08:00
033c68593b Implement a buffer pool
Downloads in quick succession could cause an OOM when the GC couldn't
  keep up
2025-01-09 02:07:36 -08:00
260a13688a Wait every 200 downloads so CV doesn't get overloaded 2024-12-26 17:50:25 -08:00
e04469938d Fix downloading images that are not hashed 2024-12-26 16:11:10 -08:00
8bdfb282b9 Only create directories if we are keeping images 2024-12-25 15:22:37 -08:00
9baa4dbc17 Add RmdirP 2024-12-25 15:11:11 -08:00
5cf25089da Improve downloading images 2024-12-25 14:53:56 -08:00
cf2c61b92e Update pre-commit 2024-12-25 14:49:22 -08:00
cc4e973bf9 Make deleting files efficient 2024-12-15 14:15:33 -08:00
7ede0dee72 Fix file closing 2024-10-31 21:12:53 -05:00
840e97cff0 Handle reading existing invalid json 2024-10-19 09:50:23 -07:00
50fcfb9513 Remove downloaded result on json failure 2024-10-19 09:41:42 -07:00
776ca68e3e Wait for page download failures 2024-10-19 09:39:08 -07:00
95fa44fb97 Fix mutex on badURLs 2024-10-18 14:35:29 -07:00
d0c7ed792a Fix infinitely increasing url length 2024-10-18 14:30:30 -07:00
97802d9111 Start cover download immediately 2024-10-18 11:48:06 -07:00
89c9b4ebce Allow truncating image files to save space 2024-10-18 11:19:50 -07:00
a65cadf106 Add CV cover downloader 2024-10-18 10:50:35 -07:00
4922ceb678 Fix locking for map storage 2024-10-16 17:56:19 -07:00
da54b3a454 Fix decoding hashes 2024-10-14 02:02:26 -07:00
f560b7f428 Fix loading empty hashes 2024-10-13 22:18:51 -07:00
87c1a69b49 Remove unused Path attribute 2024-10-13 22:16:37 -07:00
dbf03d258c Fix encoding hashes 2024-10-13 22:14:42 -07:00
095c78f0e7 Optimize iterating hashes 2024-09-12 11:42:29 -07:00
0928ed6ccf Optimize memory usage
Add a basic map storage that does manual searches to conserve memory
Change saved hashes format to allow multiple hashes for a given ID
Add a vptree storage

Maps in Go take up a huge amount of space changing IDList to []ID took
  memory from over 1GB down to 200MB (note this was on aarch64 MacOS
  which for some reason uses less memory than aarch64 Linux).
  Exhaustive searches using slices took about 30 ms search now takes
  50-60 ms as it takes longer to iterate a map. Partial hashes will
  speed up searches to 8 ms at the cost of 700MB initial memory usage
  and 400MB idle (though this is on MacOS, which for some reason uses
  less memory that aarch64 Linux so probably more like
  900MB initial -> 600 MB idle on an RPI running Linux)
2024-09-07 14:51:18 -07:00
b1de95021a Add cli flag 2024-09-02 15:35:36 -07:00
1955444dcf Add sqlite implementation 2024-09-01 18:14:19 -07:00
0069ffd5cb Make runtime hash storage modular 2024-09-01 18:13:47 -07:00
007a726764 Remove massive html file 2024-08-11 22:55:14 -07:00
d730de8fe1 Finish POC
Make a pip installable package for quick-tag
Save and load hashes from msgpack/json
Embed hashes of all thumb_urls from ComicVine
Allow associating new IDs to old IDs
Support multiple domains
Support hashing local covers
Support simple list of ids and distance and results including the hash
2024-08-11 20:46:41 -07:00
a9630ac31e Finish implementing quick_tag with simple results 2024-08-10 19:46:53 -07:00
2cbbaa0d65 Update pre-commit 2024-08-10 19:26:09 -07:00
df2906ac59 Add an option to return simple results 2024-08-05 13:54:00 -07:00
4a7e61e189 static check fixes 2024-08-04 18:26:43 -07:00
e2082465c6 more stuff 2024-08-04 18:12:00 -07:00
c61ade9961 stuff 2024-07-31 11:35:17 -07:00
4992f13ac5 add output from inital image hash comparison 2024-05-10 14:12:39 -07:00
5de0e78ee6 Add debug output for go and python 2024-05-10 13:58:04 -07:00
ce71c2f66b Initial Commit 2024-05-01 18:09:02 -07:00