lordwelch/comic-hasher
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
146 Commits 2 Branches 0 Tags
stuff
Commit Graph
146 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Timmy Welch 4b91c400b5 Add more metrics 
ids: ID Count
consistent_ids: ID Count == A Hash Count
consistent_hashes: A Hash Count == D Hash Count == P Hash Count
 2026-11-05 16:29:43 -08:00
Timmy Welch 55de1f70bf Update commands for non-blocking logging 2026-11-05 16:29:43 -08:00
Timmy Welch 7bc53e8408 Add CHDB compaction 
Compact the bolt database on shutdown to reclaim the space freed
by deleted entries
 2026-11-05 16:29:43 -08:00
Timmy Welch 978f19b1f6 Add a hash conversion command 
Add cmd/convert to convert legacy V2 hash files to the V3 format
 2026-11-05 16:29:43 -08:00
Timmy Welch 5206c6d57c Add and update tests for V3 hashes 2026-11-05 16:29:43 -08:00
Timmy Welch c98784ea1d Update storage for V3 hashes 
Backends use the V3 hashes and ids loaded in place instead of
building a separate runtime IDMap
 2026-11-05 16:29:43 -08:00
Timmy Welch b1aa8d5bd3 Serve hashes as x-image-hash-db+gzip 2026-11-05 16:29:43 -08:00
Timmy Welch 3bc88f961f Full re-scrape of comicvine 2026-11-05 16:29:43 -08:00
Timmy Welch 04c06b072a Implement the V3 binary saved hashes format 
Load hash files in place instead of decoding msgpack or json
Drop the --save-format flag since there is only one format
 2026-11-05 16:29:10 -08:00
Timmy Welch 1519a0adcb Define the SavedHashesV3 types 
V3 uses fixed-size records so a saved file can be loaded in place
and used directly instead of being decoded into new structures
 2026-11-05 16:29:08 -08:00
Timmy Welch 13e9858d92 Remove the NewIDs type 2026-11-05 16:29:06 -08:00
Timmy Welch b85659a845 Remove IDList 
Use a slice instead
 2026-11-05 16:29:01 -08:00
Timmy Welch 3cbec4606b Rename BasicMap to Basic 
Keep BasicMap as an alias for CLI flags
Fix the unknown storage type error message
 2026-11-05 16:28:59 -08:00
Timmy Welch 1129ff6f7a Rename SavedHashes to SavedHashesV2 2026-11-05 16:28:57 -08:00
Timmy Welch c21f55dac7 Improve error handling in matchCoverHash 2026-11-05 16:28:55 -08:00
Timmy Welch d720d02911 Use a bounded context for graceful shutdown 2026-11-05 16:28:53 -08:00
Timmy Welch 37d7532706 Rework the channel pipeline for hashing images 2026-11-05 16:28:51 -08:00
Timmy Welch ffc0261ace Fix HashServer 
Use appropriate digest header caches
Name fixups
Encode hashes with pgzip and a buffered writer
Add --use-all-checksums for faster default exports
Only create the hash server when hash saving is enabled
 2026-11-05 16:28:49 -08:00
Timmy Welch 132bf95205 Add tests for HashServer 2026-11-05 16:28:44 -08:00
Timmy Welch e374614124 Move hashing code into ch/ and comic-hasher to the repo root 
So `go install gitea.narnian.us/lordwelch/comic-hasher@latest` works
 2026-11-05 16:28:41 -08:00
Timmy Welch d3975a7f0b Make the max url parameter optional 
Default max to 8 when the parameter is not given
 2026-11-05 16:28:39 -08:00
Timmy Welch 8a0cf77ea4 Metron tests 2026-11-05 16:28:36 -08:00
Timmy Welch af32f6bef3 Implement metron downloader 2026-11-05 16:28:33 -08:00
Timmy Welch f66878b856 Fixup cv 2026-11-05 16:28:30 -08:00
Timmy Welch 7943f98f02 Format ids with %#v in debug output 
Print the source domain and id with %#v so debug output is
unambiguous instead of the plain String representation
 2026-11-05 16:28:28 -08:00
Timmy Welch dbd7b86cd4 Separate image downloading into a separate package 
Extract the image download worker from the ComicVine downloader

Move the image download worker out of the ComicVine downloader so
that any source can share one download path. The ComicVine downloader
now sends downloads to the shared workers and the workers send
finished downloads to the download processor.
 2026-11-05 16:28:07 -08:00
Timmy Welch 32109308d4 Intermediate between state between Fix writing hashes and V3 
Update ID Group implementation to use slices instead of maps

Uses a slice of IDs and uses the insertion index as a reference to the ID

VP-tree:
  Fix returning 0 for the distance for matches
  Fix discarding exact matches
  Explicitly mark hash slices as unused

SQLite: Decode saved hashes for debugging/testing

Fix sqlite and vptree implementations
 2026-11-05 16:28:04 -08:00
Timmy Welch 35b6e1c2d6 Add Storage tests 2026-11-05 16:28:01 -08:00
Timmy Welch ebd2b28e5a Make a generic OpenBolt wrapper 
Panic if an invalid path is given to CHDB.
Invalid means it must be relative and the first component of the path
must already exist
 2026-11-05 16:27:58 -08:00
Timmy Welch 2a03fa93c2 Migrate packages to custom log package 
Use os.Exit in place of log.Fatal
Register a -log-level flag mapping loggers onto levels
 2026-11-05 16:27:55 -08:00
Timmy Welch 15a1510cbb Add a leveled logging package 
Write through a diode queue so a slow writer never blocks the
caller
Aliases allow multiple loggers to have their level set at the same time
The "main" logger is a default normal logger "" is rewritten to "main"
The ALL pseudo-logger controls the default logger level
FormatInt formats human readable numbers
Supply a flag.Value for parsing a cli flag
logger names preserve case. Prefer functions over setting values directly
Tests are included
 2026-11-05 16:27:52 -08:00
Timmy Welch aa7bf3c2af go fmt 2026-11-02 15:19:36 -08:00
Timmy Welch 5d3bed547f Fix writing hashes 
Write to the gzip writer
Fix renaming the new file onto the old hashes
 2026-09-28 16:54:50 -07:00
Timmy Welch 5b26690994 Improve map performance 2026-09-23 12:55:43 -07:00
Timmy Welch f3e0c9f49a Fix id check 2026-09-06 16:19:27 -07:00
Timmy Welch f5e22f2e1d Stop overwriting results 2026-09-06 15:46:58 -07:00
Timmy Welch a9338fe930 Fix compression order for downloading hashes 2026-09-06 15:25:08 -07:00
Timmy Welch 60d25d43f5 Remove duplicate matches 2026-09-06 13:29:39 -07:00
Timmy Welch 9abcf09457 Use a smaller struct for basic map 
Memory savings 377MB -> 331MB
Search may be faster GC appears to be skewing results
 2026-09-06 01:42:19 -07:00
Timmy Welch c19c006057 Compress hashes 2026-09-03 13:25:08 -07:00
Timmy Welch 52a16df180 Reduce log spam 2026-09-03 05:55:14 -07:00
Timmy Welch b25fb04a46 Fix hashServer initialization 2026-09-03 04:48:27 -07:00
Timmy Welch 5f02fdecff Fix downloader failing to continue when comic counts don't match 2026-09-03 03:59:44 -07:00
Timmy Welch b4a5c0d22f Allow downloading 60 pages at a time 
Allows the full site to be scraped in a week instead of 1.5 months
 2026-09-03 03:59:44 -07:00
Timmy Welch 460c66aea0 Automatically remove the CHDB file 2026-09-03 03:56:40 -07:00
Timmy Welch acafe20bda Add metrics 2026-09-03 03:32:11 -07:00
Timmy Welch 8bcecfa307 Fix performance 2026-09-03 03:24:12 -07:00
Timmy Welch dcccbcfb60 fixup 2026-09-02 20:53:38 -07:00
Timmy Welch f7e42f3171 Fix ID's not getting added to basicmap 2026-09-02 18:42:56 -07:00
Timmy Welch 9ba007645f Rearrange logs 2026-05-17 18:29:03 -07:00