timmy
5d3bed547f
Fix writing hashes
...
Write to the gzip writer
Fix renaming the new file onto the old hashes
2026-09-28 16:54:50 -07:00
timmy
5b26690994
Improve map performance
2026-09-23 12:55:43 -07:00
timmy
f3e0c9f49a
Fix id check
2026-09-06 16:19:27 -07:00
timmy
f5e22f2e1d
Stop overwriting results
2026-09-06 15:46:58 -07:00
timmy
a9338fe930
Fix compression order for downloading hashes
2026-09-06 15:25:08 -07:00
timmy
60d25d43f5
Remove duplicate matches
2026-09-06 13:29:39 -07:00
timmy
9abcf09457
Use a smaller struct for basic map
...
Memory savings 377MB -> 331MB
Search may be faster GC appears to be skewing results
2026-09-06 01:42:19 -07:00
timmy
c19c006057
Compress hashes
2026-09-03 13:25:08 -07:00
timmy
52a16df180
Reduce log spam
2026-09-03 05:55:14 -07:00
timmy
b25fb04a46
Fix hashServer initialization
2026-09-03 04:48:27 -07:00
timmy
5f02fdecff
Fix downloader failing to continue when comic counts don't match
2026-09-03 03:59:44 -07:00
timmy
b4a5c0d22f
Allow downloading 60 pages at a time
...
Allows the full site to be scraped in a week instead of 1.5 months
2026-09-03 03:59:44 -07:00
timmy
460c66aea0
Automatically remove the CHDB file
2026-09-03 03:56:40 -07:00
timmy
acafe20bda
Add metrics
2026-09-03 03:32:11 -07:00
timmy
8bcecfa307
Fix performance
2026-09-03 03:24:12 -07:00
timmy
dcccbcfb60
fixup
2026-09-02 20:53:38 -07:00
timmy
f7e42f3171
Fix ID's not getting added to basicmap
2026-09-02 18:42:56 -07:00
timmy
9ba007645f
Rearrange logs
2026-05-17 18:29:03 -07:00
timmy
412b0e07bd
Implement downloading and automatically saving the current hashes.gz
2026-05-17 18:28:39 -07:00
timmy
a3b42127b1
Make the gokrazy build slightly smaller
2026-03-10 17:44:16 -07:00
timmy
2c8b9bc064
Update hashes from comicvine to 03-04-2026
2026-03-04 15:17:58 -08:00
timmy
ba8c162c50
go get -u
2026-03-01 16:12:44 -08:00
timmy
5c3be1e733
Fix finding exact hashes
2026-03-01 16:11:55 -08:00
timmy
7e7a7e1365
Fix missing path
2026-03-01 14:52:00 -08:00
timmy
3000ba74cc
Change default storage type
2026-03-01 14:32:08 -08:00
timmy
3d1bac490d
Remove unnecessary print
2026-03-01 14:26:50 -08:00
timmy
a3d78cd2c3
Fix download paths
...
Separate cv download functions
2026-03-01 14:04:01 -08:00
timmy
88f52c942c
Cleanup flag value after parsing
2026-02-25 18:41:18 -08:00
timmy
a6447abf1b
Use relative paths
2026-02-25 18:28:56 -08:00
timmy
2e980efa32
Ensure all cli sections are shown
2026-02-25 17:12:05 -08:00
timmy
37d279c4df
Re-add download section
2026-02-25 17:09:05 -08:00
timmy
1c2d39ea40
Make better defaults
2026-02-25 17:00:51 -08:00
timmy
197574ac67
Format bool arguments better
2026-02-25 16:21:26 -08:00
timmy
3a8225a8b4
go get -u
2026-02-25 12:09:03 -08:00
timmy
d1d4a6f2d6
Cleanup go routines
2026-02-25 12:06:14 -08:00
timmy
8cbb1ae09e
Cleanup cli options
2026-02-25 12:05:46 -08:00
timmy
211bd2b5e9
Reduce copies
2026-02-25 00:56:53 -08:00
timmy
543d94c2d4
Further reduce allocations
2026-02-24 22:50:53 -08:00
timmy
ab961da4de
Try to decode strings more efficiently
2026-02-24 22:33:43 -08:00
timmy
5a8f7deac5
Fix saving hashes
2026-02-24 22:00:01 -08:00
timmy
131e99ba82
Add --version
2026-02-22 13:55:01 -08:00
timmy
2ed5daa67b
Cleanup
2026-02-22 13:49:03 -08:00
timmy
e4f3cc6008
Remove cover_extract
2026-02-22 13:41:17 -08:00
timmy
e82794e459
update pre-commit
2026-02-22 12:30:50 -08:00
timmy
32ec6147ea
More logging
2026-02-22 12:18:28 -08:00
timmy
22d59aa221
Move HashStorage to its own package
2025-05-31 19:00:40 -07:00
timmy
ed0b5ba441
Don't include sqlite/vptree in gokrazy build
2025-05-31 18:35:18 -07:00
timmy
d240bd953b
Remove references to loaded json object
2025-05-31 17:59:19 -07:00
timmy
bb64d8449c
Improve memory efficiency
2025-05-31 15:00:05 -07:00
timmy
b0a4c2939c
Map to slice
2025-05-27 22:31:11 -07:00
timmy
3e364b1858
Filter out non-json files
2025-04-24 12:54:40 -07:00
timmy
0da76f7fb5
Fix inserting downloaded pages
2025-04-24 12:53:55 -07:00
timmy
d7c42f5c1d
Fix issues with loading saved hashes
2025-04-24 12:46:04 -07:00
timmy
374c46bc48
Fix url handling order
2025-04-24 12:46:04 -07:00
timmy
aca658e32d
Typo
2025-04-24 12:46:04 -07:00
timmy
acd71df302
Add the issue number to the downloaded CV pages
...
Update existing pages 10 at a time if they are missing data
Simplify checking for previously downloaded pages
2025-04-24 12:46:04 -07:00
timmy
0fd431f6f7
Update result type
2025-04-19 15:41:04 -07:00
timmy
9322f754bf
Fix http server
2025-04-13 16:42:49 -07:00
timmy
486cf074d0
Remove replace directive
2025-04-13 16:07:49 -07:00
timmy
bf23bb5a4c
Add Server http Header and add -listen and -debug options
2025-04-13 15:58:34 -07:00
timmy
d7946c2aaf
Fix simple results
2025-04-13 15:57:29 -07:00
timmy
f54b4b1d9d
Fix exact matches to match previous behavior
2025-04-13 15:54:17 -07:00
timmy
a2765b0582
Re-implement sqlite
2025-02-23 12:31:23 -08:00
timmy
1f37684862
Fix loading saved hashes
2025-02-23 12:31:09 -08:00
timmy
5307b5d8df
Update pre-commit
2025-02-23 12:29:23 -08:00
timmy
130b7dec4a
Update internal hash storage
2025-02-22 13:45:41 -08:00
timmy
6452f2e50d
Update hashes from comicvine to 02-06-2025
2025-02-06 16:19:05 -08:00
timmy
f52219cb31
Fix loading old hashes
2025-02-05 17:59:59 -08:00
timmy
5a93dacdad
Fix checking for new covers
2025-02-05 16:32:25 -08:00
timmy
798ae49c8b
Update dependencies
2025-01-29 16:53:47 -08:00
timmy
8d6db630d4
Fix channel initialization
2025-01-21 18:26:12 -08:00
timmy
29f58e7fe7
Limit download buffer pool to 10MB buffers
2025-01-11 16:15:36 -08:00
timmy
fe3f045c6e
Fix issues with reading cv files
2025-01-11 15:26:35 -08:00
timmy
8ce1ca3354
Fix invalid index
2025-01-11 12:19:14 -08:00
timmy
75d60339ee
Optimizations
...
Re-arrange some channels
set all channels to a size of 1
Reset the image download buffer
Allocate the correct slice size for the CV API
Add a flag to only download the original image
2025-01-10 22:28:51 -08:00
timmy
8a9aec4884
Don't catch sigabrt
2025-01-10 14:24:12 -08:00
timmy
033c68593b
Implement a buffer pool
...
Downloads in quick succession could cause an OOM when the GC couldn't
keep up
2025-01-09 02:07:36 -08:00
timmy
260a13688a
Wait every 200 downloads so CV doesn't get overloaded
2024-12-26 17:50:25 -08:00
timmy
e04469938d
Fix downloading images that are not hashed
2024-12-26 16:11:10 -08:00
timmy
8bdfb282b9
Only create directories if we are keeping images
2024-12-25 15:22:37 -08:00
timmy
9baa4dbc17
Add RmdirP
2024-12-25 15:11:11 -08:00
timmy
5cf25089da
Improve downloading images
2024-12-25 14:53:56 -08:00
timmy
cf2c61b92e
Update pre-commit
2024-12-25 14:49:22 -08:00
timmy
cc4e973bf9
Make deleting files efficient
2024-12-15 14:15:33 -08:00
timmy
7ede0dee72
Fix file closing
2024-10-31 21:12:53 -05:00
timmy
840e97cff0
Handle reading existing invalid json
2024-10-19 09:50:23 -07:00
timmy
50fcfb9513
Remove downloaded result on json failure
2024-10-19 09:41:42 -07:00
timmy
776ca68e3e
Wait for page download failures
2024-10-19 09:39:08 -07:00
timmy
95fa44fb97
Fix mutex on badURLs
2024-10-18 14:35:29 -07:00
timmy
d0c7ed792a
Fix infinitely increasing url length
2024-10-18 14:30:30 -07:00
timmy
97802d9111
Start cover download immediately
2024-10-18 11:48:06 -07:00
timmy
89c9b4ebce
Allow truncating image files to save space
2024-10-18 11:19:50 -07:00
timmy
a65cadf106
Add CV cover downloader
2024-10-18 10:50:35 -07:00
timmy
4922ceb678
Fix locking for map storage
2024-10-16 17:56:19 -07:00
timmy
da54b3a454
Fix decoding hashes
2024-10-14 02:02:26 -07:00
timmy
f560b7f428
Fix loading empty hashes
2024-10-13 22:18:51 -07:00
timmy
87c1a69b49
Remove unused Path attribute
2024-10-13 22:16:37 -07:00
timmy
dbf03d258c
Fix encoding hashes
2024-10-13 22:14:42 -07:00
timmy
095c78f0e7
Optimize iterating hashes
2024-09-12 11:42:29 -07:00
timmy
0928ed6ccf
Optimize memory usage
...
Add a basic map storage that does manual searches to conserve memory
Change saved hashes format to allow multiple hashes for a given ID
Add a vptree storage
Maps in Go take up a huge amount of space changing IDList to []ID took
memory from over 1GB down to 200MB (note this was on aarch64 MacOS
which for some reason uses less memory than aarch64 Linux).
Exhaustive searches using slices took about 30 ms search now takes
50-60 ms as it takes longer to iterate a map. Partial hashes will
speed up searches to 8 ms at the cost of 700MB initial memory usage
and 400MB idle (though this is on MacOS, which for some reason uses
less memory that aarch64 Linux so probably more like
900MB initial -> 600 MB idle on an RPI running Linux)
2024-09-07 14:51:18 -07:00
timmy