Timmy Welch
b85659a845
Remove IDList
...
Use a slice instead
2026-11-05 16:29:01 -08:00
Timmy Welch
e374614124
Move hashing code into ch/ and comic-hasher to the repo root
...
So `go install gitea.narnian.us/lordwelch/comic-hasher@latest` works
2026-11-05 16:28:41 -08:00
Timmy Welch
f66878b856
Fixup cv
2026-11-05 16:28:30 -08:00
Timmy Welch
dbd7b86cd4
Separate image downloading into a separate package
...
Extract the image download worker from the ComicVine downloader
Move the image download worker out of the ComicVine downloader so
that any source can share one download path. The ComicVine downloader
now sends downloads to the shared workers and the workers send
finished downloads to the download processor.
2026-11-05 16:28:07 -08:00
Timmy Welch
2a03fa93c2
Migrate packages to custom log package
...
Use os.Exit in place of log.Fatal
Register a -log-level flag mapping loggers onto levels
2026-11-05 16:27:55 -08:00
Timmy Welch
52a16df180
Reduce log spam
2026-09-03 05:55:14 -07:00
Timmy Welch
5f02fdecff
Fix downloader failing to continue when comic counts don't match
2026-09-03 03:59:44 -07:00
Timmy Welch
b4a5c0d22f
Allow downloading 60 pages at a time
...
Allows the full site to be scraped in a week instead of 1.5 months
2026-09-03 03:59:44 -07:00
Timmy Welch
f7e42f3171
Fix ID's not getting added to basicmap
2026-09-02 18:42:56 -07:00
Timmy Welch
a3d78cd2c3
Fix download paths
...
Separate cv download functions
2026-03-01 14:04:01 -08:00
Timmy Welch
88f52c942c
Cleanup flag value after parsing
2026-02-25 18:41:18 -08:00
Timmy Welch
a6447abf1b
Use relative paths
2026-02-25 18:28:56 -08:00
Timmy Welch
d1d4a6f2d6
Cleanup go routines
2026-02-25 12:06:14 -08:00
Timmy Welch
8cbb1ae09e
Cleanup cli options
2026-02-25 12:05:46 -08:00
Timmy Welch
2ed5daa67b
Cleanup
2026-02-22 13:49:03 -08:00
Timmy Welch
bb64d8449c
Improve memory efficiency
2025-05-31 15:00:05 -07:00
Timmy Welch
3e364b1858
Filter out non-json files
2025-04-24 12:54:40 -07:00
Timmy Welch
0da76f7fb5
Fix inserting downloaded pages
2025-04-24 12:53:55 -07:00
Timmy Welch
374c46bc48
Fix url handling order
2025-04-24 12:46:04 -07:00
Timmy Welch
acd71df302
Add the issue number to the downloaded CV pages
...
Update existing pages 10 at a time if they are missing data
Simplify checking for previously downloaded pages
2025-04-24 12:46:04 -07:00
Timmy Welch
130b7dec4a
Update internal hash storage
2025-02-22 13:45:41 -08:00
Timmy Welch
5a93dacdad
Fix checking for new covers
2025-02-05 16:32:25 -08:00
Timmy Welch
8d6db630d4
Fix channel initialization
2025-01-21 18:26:12 -08:00
Timmy Welch
29f58e7fe7
Limit download buffer pool to 10MB buffers
2025-01-11 16:15:36 -08:00
Timmy Welch
fe3f045c6e
Fix issues with reading cv files
2025-01-11 15:26:35 -08:00
Timmy Welch
8ce1ca3354
Fix invalid index
2025-01-11 12:19:14 -08:00
Timmy Welch
75d60339ee
Optimizations
...
Re-arrange some channels
set all channels to a size of 1
Reset the image download buffer
Allocate the correct slice size for the CV API
Add a flag to only download the original image
2025-01-10 22:28:51 -08:00
Timmy Welch
033c68593b
Implement a buffer pool
...
Downloads in quick succession could cause an OOM when the GC couldn't
keep up
2025-01-09 02:07:36 -08:00
Timmy Welch
260a13688a
Wait every 200 downloads so CV doesn't get overloaded
2024-12-26 17:50:25 -08:00
Timmy Welch
8bdfb282b9
Only create directories if we are keeping images
2024-12-25 15:22:37 -08:00
Timmy Welch
5cf25089da
Improve downloading images
2024-12-25 14:53:56 -08:00
Timmy Welch
cc4e973bf9
Make deleting files efficient
2024-12-15 14:15:33 -08:00
Timmy Welch
7ede0dee72
Fix file closing
2024-10-31 21:12:53 -05:00
Timmy Welch
840e97cff0
Handle reading existing invalid json
2024-10-19 09:50:23 -07:00
Timmy Welch
50fcfb9513
Remove downloaded result on json failure
2024-10-19 09:41:42 -07:00
Timmy Welch
776ca68e3e
Wait for page download failures
2024-10-19 09:39:08 -07:00
Timmy Welch
95fa44fb97
Fix mutex on badURLs
2024-10-18 14:35:29 -07:00
Timmy Welch
d0c7ed792a
Fix infinitely increasing url length
2024-10-18 14:30:30 -07:00
Timmy Welch
a65cadf106
Add CV cover downloader
2024-10-18 10:50:35 -07:00
Timmy Welch