Timmy Welch
8a9aec4884
Don't catch sigabrt
2025-01-10 14:24:12 -08:00
Timmy Welch
033c68593b
Implement a buffer pool
...
Downloads in quick succession could cause an OOM when the GC couldn't
keep up
2025-01-09 02:07:36 -08:00
Timmy Welch
260a13688a
Wait every 200 downloads so CV doesn't get overloaded
2024-12-26 17:50:25 -08:00
Timmy Welch
e04469938d
Fix downloading images that are not hashed
2024-12-26 16:11:10 -08:00
Timmy Welch
8bdfb282b9
Only create directories if we are keeping images
2024-12-25 15:22:37 -08:00
Timmy Welch
9baa4dbc17
Add RmdirP
2024-12-25 15:11:11 -08:00
Timmy Welch
5cf25089da
Improve downloading images
2024-12-25 14:53:56 -08:00
Timmy Welch
cf2c61b92e
Update pre-commit
2024-12-25 14:49:22 -08:00
Timmy Welch
cc4e973bf9
Make deleting files efficient
2024-12-15 14:15:33 -08:00
Timmy Welch
7ede0dee72
Fix file closing
2024-10-31 21:12:53 -05:00
Timmy Welch
840e97cff0
Handle reading existing invalid json
2024-10-19 09:50:23 -07:00
Timmy Welch
50fcfb9513
Remove downloaded result on json failure
2024-10-19 09:41:42 -07:00
Timmy Welch
776ca68e3e
Wait for page download failures
2024-10-19 09:39:08 -07:00
Timmy Welch
95fa44fb97
Fix mutex on badURLs
2024-10-18 14:35:29 -07:00
Timmy Welch
d0c7ed792a
Fix infinitely increasing url length
2024-10-18 14:30:30 -07:00
Timmy Welch
97802d9111
Start cover download immediately
2024-10-18 11:48:06 -07:00
Timmy Welch
89c9b4ebce
Allow truncating image files to save space
2024-10-18 11:19:50 -07:00
Timmy Welch
a65cadf106
Add CV cover downloader
2024-10-18 10:50:35 -07:00
Timmy Welch
4922ceb678
Fix locking for map storage
2024-10-16 17:56:19 -07:00
Timmy Welch
da54b3a454
Fix decoding hashes
2024-10-14 02:02:26 -07:00
Timmy Welch
f560b7f428
Fix loading empty hashes
2024-10-13 22:18:51 -07:00
Timmy Welch
87c1a69b49
Remove unused Path attribute
2024-10-13 22:16:37 -07:00
Timmy Welch
dbf03d258c
Fix encoding hashes
2024-10-13 22:14:42 -07:00
Timmy Welch
095c78f0e7
Optimize iterating hashes
2024-09-12 11:42:29 -07:00
Timmy Welch
0928ed6ccf
Optimize memory usage
...
Add a basic map storage that does manual searches to conserve memory
Change saved hashes format to allow multiple hashes for a given ID
Add a vptree storage
Maps in Go take up a huge amount of space changing IDList to []ID took
memory from over 1GB down to 200MB (note this was on aarch64 MacOS
which for some reason uses less memory than aarch64 Linux).
Exhaustive searches using slices took about 30 ms search now takes
50-60 ms as it takes longer to iterate a map. Partial hashes will
speed up searches to 8 ms at the cost of 700MB initial memory usage
and 400MB idle (though this is on MacOS, which for some reason uses
less memory that aarch64 Linux so probably more like
900MB initial -> 600 MB idle on an RPI running Linux)
2024-09-07 14:51:18 -07:00
Timmy Welch
b1de95021a
Add cli flag
2024-09-02 15:35:36 -07:00
Timmy Welch
1955444dcf
Add sqlite implementation
2024-09-01 18:14:19 -07:00
Timmy Welch
0069ffd5cb
Make runtime hash storage modular
2024-09-01 18:13:47 -07:00
Timmy Welch
007a726764
Remove massive html file
2024-08-11 22:55:14 -07:00
Timmy Welch
d730de8fe1
Finish POC
...
Make a pip installable package for quick-tag
Save and load hashes from msgpack/json
Embed hashes of all thumb_urls from ComicVine
Allow associating new IDs to old IDs
Support multiple domains
Support hashing local covers
Support simple list of ids and distance and results including the hash
2024-08-11 20:46:41 -07:00
Timmy Welch
a9630ac31e
Finish implementing quick_tag with simple results
2024-08-10 19:46:53 -07:00
Timmy Welch
2cbbaa0d65
Update pre-commit
2024-08-10 19:26:09 -07:00
Timmy Welch
df2906ac59
Add an option to return simple results
2024-08-05 13:54:00 -07:00
Timmy Welch
4a7e61e189
static check fixes
2024-08-04 18:26:43 -07:00
Timmy Welch
e2082465c6
more stuff
2024-08-04 18:12:00 -07:00
Timmy Welch
c61ade9961
stuff
2024-07-31 11:35:17 -07:00
Timmy Welch
4992f13ac5
add output from inital image hash comparison
2024-05-10 14:12:39 -07:00
Timmy Welch
5de0e78ee6
Add debug output for go and python
2024-05-10 13:58:04 -07:00
Timmy Welch
ce71c2f66b
Initial Commit
2024-05-01 18:09:02 -07:00