lordwelch/comic-hasher
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
146 Commits 2 Branches 0 Tags
stuff
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Timmy Welch 4b91c400b5 Add more metrics 
ids: ID Count
consistent_ids: ID Count == A Hash Count
consistent_hashes: A Hash Count == D Hash Count == P Hash Count
 2026-11-05 16:29:43 -08:00
Timmy Welch 55de1f70bf Update commands for non-blocking logging 2026-11-05 16:29:43 -08:00
Timmy Welch 7bc53e8408 Add CHDB compaction 
Compact the bolt database on shutdown to reclaim the space freed
by deleted entries
 2026-11-05 16:29:43 -08:00
Timmy Welch 978f19b1f6 Add a hash conversion command 
Add cmd/convert to convert legacy V2 hash files to the V3 format
 2026-11-05 16:29:43 -08:00
Timmy Welch 5206c6d57c Add and update tests for V3 hashes 2026-11-05 16:29:43 -08:00
Timmy Welch c98784ea1d Update storage for V3 hashes 
Backends use the V3 hashes and ids loaded in place instead of
building a separate runtime IDMap
 2026-11-05 16:29:43 -08:00
Timmy Welch b1aa8d5bd3 Serve hashes as x-image-hash-db+gzip 2026-11-05 16:29:43 -08:00
Timmy Welch 3bc88f961f Full re-scrape of comicvine 2026-11-05 16:29:43 -08:00
Timmy Welch 04c06b072a Implement the V3 binary saved hashes format 
Load hash files in place instead of decoding msgpack or json
Drop the --save-format flag since there is only one format
 2026-11-05 16:29:10 -08:00
Timmy Welch 1519a0adcb Define the SavedHashesV3 types 
V3 uses fixed-size records so a saved file can be loaded in place
and used directly instead of being decoded into new structures
 2026-11-05 16:29:08 -08:00
Timmy Welch 13e9858d92 Remove the NewIDs type 2026-11-05 16:29:06 -08:00
Timmy Welch b85659a845 Remove IDList 
Use a slice instead
 2026-11-05 16:29:01 -08:00
Timmy Welch 3cbec4606b Rename BasicMap to Basic 
Keep BasicMap as an alias for CLI flags
Fix the unknown storage type error message
 2026-11-05 16:28:59 -08:00
Timmy Welch 1129ff6f7a Rename SavedHashes to SavedHashesV2 2026-11-05 16:28:57 -08:00
Timmy Welch c21f55dac7 Improve error handling in matchCoverHash 2026-11-05 16:28:55 -08:00
Timmy Welch d720d02911 Use a bounded context for graceful shutdown 2026-11-05 16:28:53 -08:00
Timmy Welch 37d7532706 Rework the channel pipeline for hashing images 2026-11-05 16:28:51 -08:00
Timmy Welch ffc0261ace Fix HashServer 
Use appropriate digest header caches
Name fixups
Encode hashes with pgzip and a buffered writer
Add --use-all-checksums for faster default exports
Only create the hash server when hash saving is enabled
 2026-11-05 16:28:49 -08:00
Timmy Welch 132bf95205 Add tests for HashServer 2026-11-05 16:28:44 -08:00
Timmy Welch e374614124 Move hashing code into ch/ and comic-hasher to the repo root 
So `go install gitea.narnian.us/lordwelch/comic-hasher@latest` works
 2026-11-05 16:28:41 -08:00
Timmy Welch d3975a7f0b Make the max url parameter optional 
Default max to 8 when the parameter is not given
 2026-11-05 16:28:39 -08:00
Timmy Welch 8a0cf77ea4 Metron tests 2026-11-05 16:28:36 -08:00
Timmy Welch af32f6bef3 Implement metron downloader 2026-11-05 16:28:33 -08:00
Timmy Welch f66878b856 Fixup cv 2026-11-05 16:28:30 -08:00
Timmy Welch 7943f98f02 Format ids with %#v in debug output 
Print the source domain and id with %#v so debug output is
unambiguous instead of the plain String representation
 2026-11-05 16:28:28 -08:00
Timmy Welch dbd7b86cd4 Separate image downloading into a separate package 
Extract the image download worker from the ComicVine downloader

Move the image download worker out of the ComicVine downloader so
that any source can share one download path. The ComicVine downloader
now sends downloads to the shared workers and the workers send
finished downloads to the download processor.
 2026-11-05 16:28:07 -08:00
Timmy Welch 32109308d4 Intermediate between state between Fix writing hashes and V3 
Update ID Group implementation to use slices instead of maps

Uses a slice of IDs and uses the insertion index as a reference to the ID

VP-tree:
  Fix returning 0 for the distance for matches
  Fix discarding exact matches
  Explicitly mark hash slices as unused

SQLite: Decode saved hashes for debugging/testing

Fix sqlite and vptree implementations
 2026-11-05 16:28:04 -08:00
Timmy Welch 35b6e1c2d6 Add Storage tests 2026-11-05 16:28:01 -08:00
Timmy Welch ebd2b28e5a Make a generic OpenBolt wrapper 
Panic if an invalid path is given to CHDB.
Invalid means it must be relative and the first component of the path
must already exist
 2026-11-05 16:27:58 -08:00
Timmy Welch 2a03fa93c2 Migrate packages to custom log package 
Use os.Exit in place of log.Fatal
Register a -log-level flag mapping loggers onto levels
 2026-11-05 16:27:55 -08:00
Timmy Welch 15a1510cbb Add a leveled logging package 
Write through a diode queue so a slow writer never blocks the
caller
Aliases allow multiple loggers to have their level set at the same time
The "main" logger is a default normal logger "" is rewritten to "main"
The ALL pseudo-logger controls the default logger level
FormatInt formats human readable numbers
Supply a flag.Value for parsing a cli flag
logger names preserve case. Prefer functions over setting values directly
Tests are included
 2026-11-05 16:27:52 -08:00
Timmy Welch aa7bf3c2af go fmt 2026-11-02 15:19:36 -08:00
Timmy Welch 5d3bed547f Fix writing hashes 
Write to the gzip writer
Fix renaming the new file onto the old hashes
 2026-09-28 16:54:50 -07:00
Timmy Welch 5b26690994 Improve map performance 2026-09-23 12:55:43 -07:00
Timmy Welch f3e0c9f49a Fix id check 2026-09-06 16:19:27 -07:00
Timmy Welch f5e22f2e1d Stop overwriting results 2026-09-06 15:46:58 -07:00
Timmy Welch a9338fe930 Fix compression order for downloading hashes 2026-09-06 15:25:08 -07:00
Timmy Welch 60d25d43f5 Remove duplicate matches 2026-09-06 13:29:39 -07:00
Timmy Welch 9abcf09457 Use a smaller struct for basic map 
Memory savings 377MB -> 331MB
Search may be faster GC appears to be skewing results
 2026-09-06 01:42:19 -07:00
Timmy Welch c19c006057 Compress hashes 2026-09-03 13:25:08 -07:00
Timmy Welch 52a16df180 Reduce log spam 2026-09-03 05:55:14 -07:00
Timmy Welch b25fb04a46 Fix hashServer initialization 2026-09-03 04:48:27 -07:00
Timmy Welch 5f02fdecff Fix downloader failing to continue when comic counts don't match 2026-09-03 03:59:44 -07:00
Timmy Welch b4a5c0d22f Allow downloading 60 pages at a time 
Allows the full site to be scraped in a week instead of 1.5 months
 2026-09-03 03:59:44 -07:00
Timmy Welch 460c66aea0 Automatically remove the CHDB file 2026-09-03 03:56:40 -07:00
Timmy Welch acafe20bda Add metrics 2026-09-03 03:32:11 -07:00
Timmy Welch 8bcecfa307 Fix performance 2026-09-03 03:24:12 -07:00
Timmy Welch dcccbcfb60 fixup 2026-09-02 20:53:38 -07:00
Timmy Welch f7e42f3171 Fix ID's not getting added to basicmap 2026-09-02 18:42:56 -07:00
Timmy Welch 9ba007645f Rearrange logs 2026-05-17 18:29:03 -07:00
Timmy Welch 412b0e07bd Implement downloading and automatically saving the current hashes.gz 2026-05-17 18:28:39 -07:00
Timmy Welch a3b42127b1 Make the gokrazy build slightly smaller 2026-03-10 17:44:16 -07:00
Timmy Welch 2c8b9bc064 Update hashes from comicvine to 03-04-2026 2026-03-04 15:17:58 -08:00
Timmy Welch ba8c162c50 go get -u 2026-03-01 16:12:44 -08:00
Timmy Welch 5c3be1e733 Fix finding exact hashes 2026-03-01 16:11:55 -08:00
Timmy Welch 7e7a7e1365 Fix missing path 2026-03-01 14:52:00 -08:00
Timmy Welch 3000ba74cc Change default storage type 2026-03-01 14:32:08 -08:00
Timmy Welch 3d1bac490d Remove unnecessary print 2026-03-01 14:26:50 -08:00
Timmy Welch a3d78cd2c3 Fix download paths 
Separate cv download functions
 2026-03-01 14:04:01 -08:00
Timmy Welch 88f52c942c Cleanup flag value after parsing 2026-02-25 18:41:18 -08:00
Timmy Welch a6447abf1b Use relative paths 2026-02-25 18:28:56 -08:00
Timmy Welch 2e980efa32 Ensure all cli sections are shown 2026-02-25 17:12:05 -08:00
Timmy Welch 37d279c4df Re-add download section 2026-02-25 17:09:05 -08:00
Timmy Welch 1c2d39ea40 Make better defaults 2026-02-25 17:00:51 -08:00
Timmy Welch 197574ac67 Format bool arguments better 2026-02-25 16:21:26 -08:00
Timmy Welch 3a8225a8b4 go get -u 2026-02-25 12:09:03 -08:00
Timmy Welch d1d4a6f2d6 Cleanup go routines 2026-02-25 12:06:14 -08:00
Timmy Welch 8cbb1ae09e Cleanup cli options 2026-02-25 12:05:46 -08:00
Timmy Welch 211bd2b5e9 Reduce copies 2026-02-25 00:56:53 -08:00
Timmy Welch 543d94c2d4 Further reduce allocations 2026-02-24 22:50:53 -08:00
Timmy Welch ab961da4de Try to decode strings more efficiently 2026-02-24 22:33:43 -08:00
Timmy Welch 5a8f7deac5 Fix saving hashes 2026-02-24 22:00:01 -08:00
Timmy Welch 131e99ba82 Add --version 2026-02-22 13:55:01 -08:00
Timmy Welch 2ed5daa67b Cleanup 2026-02-22 13:49:03 -08:00
Timmy Welch e4f3cc6008 Remove cover_extract 2026-02-22 13:41:17 -08:00
Timmy Welch e82794e459 update pre-commit 2026-02-22 12:30:50 -08:00
Timmy Welch 32ec6147ea More logging 2026-02-22 12:18:28 -08:00
Timmy Welch 22d59aa221 Move HashStorage to its own package 2025-05-31 19:00:40 -07:00
Timmy Welch ed0b5ba441 Don't include sqlite/vptree in gokrazy build 2025-05-31 18:35:18 -07:00
Timmy Welch d240bd953b Remove references to loaded json object 2025-05-31 17:59:19 -07:00
Timmy Welch bb64d8449c Improve memory efficiency 2025-05-31 15:00:05 -07:00
Timmy Welch b0a4c2939c Map to slice 2025-05-27 22:31:11 -07:00
Timmy Welch 3e364b1858 Filter out non-json files 2025-04-24 12:54:40 -07:00
Timmy Welch 0da76f7fb5 Fix inserting downloaded pages 2025-04-24 12:53:55 -07:00
Timmy Welch d7c42f5c1d Fix issues with loading saved hashes 2025-04-24 12:46:04 -07:00
Timmy Welch 374c46bc48 Fix url handling order 2025-04-24 12:46:04 -07:00
Timmy Welch aca658e32d Typo 2025-04-24 12:46:04 -07:00
Timmy Welch acd71df302 Add the issue number to the downloaded CV pages 
Update existing pages 10 at a time if they are missing data
Simplify checking for previously downloaded pages
 2025-04-24 12:46:04 -07:00
Timmy Welch 0fd431f6f7 Update result type 2025-04-19 15:41:04 -07:00
Timmy Welch 9322f754bf Fix http server 2025-04-13 16:42:49 -07:00
Timmy Welch 486cf074d0 Remove replace directive 2025-04-13 16:07:49 -07:00
Timmy Welch bf23bb5a4c Add Server http Header and add -listen and -debug options 2025-04-13 15:58:34 -07:00
Timmy Welch d7946c2aaf Fix simple results 2025-04-13 15:57:29 -07:00
Timmy Welch f54b4b1d9d Fix exact matches to match previous behavior 2025-04-13 15:54:17 -07:00
Timmy Welch a2765b0582 Re-implement sqlite 2025-02-23 12:31:23 -08:00
Timmy Welch 1f37684862 Fix loading saved hashes 2025-02-23 12:31:09 -08:00
Timmy Welch 5307b5d8df Update pre-commit 2025-02-23 12:29:23 -08:00
Timmy Welch 130b7dec4a Update internal hash storage 2025-02-22 13:45:41 -08:00
Timmy Welch 6452f2e50d Update hashes from comicvine to 02-06-2025 2025-02-06 16:19:05 -08:00
Timmy Welch f52219cb31 Fix loading old hashes 2025-02-05 17:59:59 -08:00