23 Commits
Author SHA1 Message Date
Timmy Welch 978f19b1f6 Add a hash conversion command
Add cmd/convert to convert legacy V2 hash files to the V3 format
2026-11-05 16:29:43 -08:00
Timmy Welch 04c06b072a Implement the V3 binary saved hashes format
Load hash files in place instead of decoding msgpack or json
Drop the --save-format flag since there is only one format
2026-11-05 16:29:10 -08:00
Timmy Welch ffc0261ace Fix HashServer
Use appropriate digest header caches
Name fixups
Encode hashes with pgzip and a buffered writer
Add --use-all-checksums for faster default exports
Only create the hash server when hash saving is enabled
2026-11-05 16:28:49 -08:00
Timmy Welch af32f6bef3 Implement metron downloader 2026-11-05 16:28:33 -08:00
Timmy Welch 15a1510cbb Add a leveled logging package
Write through a diode queue so a slow writer never blocks the
caller
Aliases allow multiple loggers to have their level set at the same time
The "main" logger is a default normal logger "" is rewritten to "main"
The ALL pseudo-logger controls the default logger level
FormatInt formats human readable numbers
Supply a flag.Value for parsing a cli flag
logger names preserve case. Prefer functions over setting values directly
Tests are included
2026-11-05 16:27:52 -08:00
Timmy Welch acafe20bda Add metrics 2026-09-03 03:32:11 -07:00
Timmy Welch 412b0e07bd Implement downloading and automatically saving the current hashes.gz 2026-05-17 18:28:39 -07:00
Timmy Welch ba8c162c50 go get -u 2026-03-01 16:12:44 -08:00
Timmy Welch 3a8225a8b4 go get -u 2026-02-25 12:09:03 -08:00
Timmy Welch 5a8f7deac5 Fix saving hashes 2026-02-24 22:00:01 -08:00
Timmy Welch e82794e459 update pre-commit 2026-02-22 12:30:50 -08:00
Timmy Welch ed0b5ba441 Don't include sqlite/vptree in gokrazy build 2025-05-31 18:35:18 -07:00
Timmy Welch b0a4c2939c Map to slice 2025-05-27 22:31:11 -07:00
Timmy Welch 130b7dec4a Update internal hash storage 2025-02-22 13:45:41 -08:00
Timmy Welch 798ae49c8b Update dependencies 2025-01-29 16:53:47 -08:00
Timmy Welch 0928ed6ccf Optimize memory usage
Add a basic map storage that does manual searches to conserve memory
Change saved hashes format to allow multiple hashes for a given ID
Add a vptree storage

Maps in Go take up a huge amount of space changing IDList to []ID took
  memory from over 1GB down to 200MB (note this was on aarch64 MacOS
  which for some reason uses less memory than aarch64 Linux).
  Exhaustive searches using slices took about 30 ms search now takes
  50-60 ms as it takes longer to iterate a map. Partial hashes will
  speed up searches to 8 ms at the cost of 700MB initial memory usage
  and 400MB idle (though this is on MacOS, which for some reason uses
  less memory that aarch64 Linux so probably more like
  900MB initial -> 600 MB idle on an RPI running Linux)
2024-09-07 14:51:18 -07:00
Timmy Welch 1955444dcf Add sqlite implementation 2024-09-01 18:14:19 -07:00
Timmy Welch 0069ffd5cb Make runtime hash storage modular 2024-09-01 18:13:47 -07:00
Timmy Welch d730de8fe1 Finish POC
Make a pip installable package for quick-tag
Save and load hashes from msgpack/json
Embed hashes of all thumb_urls from ComicVine
Allow associating new IDs to old IDs
Support multiple domains
Support hashing local covers
Support simple list of ids and distance and results including the hash
2024-08-11 20:46:41 -07:00
Timmy Welch e2082465c6 more stuff 2024-08-04 18:12:00 -07:00
Timmy Welch c61ade9961 stuff 2024-07-31 11:35:17 -07:00
Timmy Welch 5de0e78ee6 Add debug output for go and python 2024-05-10 13:58:04 -07:00
Timmy Welch ce71c2f66b Initial Commit 2024-05-01 18:09:02 -07:00