0928ed6ccf
Add a basic map storage that does manual searches to conserve memory Change saved hashes format to allow multiple hashes for a given ID Add a vptree storage Maps in Go take up a huge amount of space changing IDList to []ID took memory from over 1GB down to 200MB (note this was on aarch64 MacOS which for some reason uses less memory than aarch64 Linux). Exhaustive searches using slices took about 30 ms search now takes 50-60 ms as it takes longer to iterate a map. Partial hashes will speed up searches to 8 ms at the cost of 700MB initial memory usage and 400MB idle (though this is on MacOS, which for some reason uses less memory that aarch64 Linux so probably more like 900MB initial -> 600 MB idle on an RPI running Linux)
62 lines
2.4 KiB
Modula-2
62 lines
2.4 KiB
Modula-2
module gitea.narnian.us/lordwelch/comic-hasher
|
|
|
|
go 1.23.0
|
|
|
|
require (
|
|
gitea.narnian.us/lordwelch/goimagehash v0.0.0-20240812025715-33ff96e45f00
|
|
github.com/fmartingr/go-comicinfo/v2 v2.0.2
|
|
github.com/kr/pretty v0.1.0
|
|
github.com/mattn/go-sqlite3 v1.14.22
|
|
github.com/mholt/archiver/v4 v4.0.0-alpha.8
|
|
github.com/ncruces/go-sqlite3 v0.18.1
|
|
golang.org/x/image v0.19.0
|
|
golang.org/x/text v0.17.0
|
|
gonum.org/v1/gonum v0.15.1
|
|
modernc.org/sqlite v1.32.0
|
|
)
|
|
|
|
require (
|
|
github.com/vmihailenco/msgpack/v5 v5.4.1
|
|
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
|
|
)
|
|
|
|
require (
|
|
github.com/andybalholm/brotli v1.0.4 // indirect
|
|
github.com/bodgit/plumbing v1.2.0 // indirect
|
|
github.com/bodgit/sevenzip v1.3.0 // indirect
|
|
github.com/bodgit/windows v1.0.0 // indirect
|
|
github.com/connesc/cipherio v0.2.1 // indirect
|
|
github.com/disintegration/imaging v1.6.3-0.20201218193011-d40f48ce0f09 // indirect
|
|
github.com/dsnet/compress v0.0.1 // indirect
|
|
github.com/dustin/go-humanize v1.0.1 // indirect
|
|
github.com/golang/mock v1.6.0 // indirect
|
|
github.com/golang/snappy v0.0.4 // indirect
|
|
github.com/google/uuid v1.6.0 // indirect
|
|
github.com/hashicorp/errwrap v1.0.0 // indirect
|
|
github.com/hashicorp/go-multierror v1.1.1 // indirect
|
|
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
|
|
github.com/klauspost/compress v1.15.9 // indirect
|
|
github.com/klauspost/pgzip v1.2.5 // indirect
|
|
github.com/kr/text v0.1.0 // indirect
|
|
github.com/mattn/go-isatty v0.0.20 // indirect
|
|
github.com/ncruces/go-strftime v0.1.9 // indirect
|
|
github.com/ncruces/julianday v1.0.0 // indirect
|
|
github.com/nwaples/rardecode/v2 v2.0.0-beta.2 // indirect
|
|
github.com/pierrec/lz4/v4 v4.1.15 // indirect
|
|
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
|
github.com/tetratelabs/wazero v1.8.0 // indirect
|
|
github.com/therootcompany/xz v1.0.1 // indirect
|
|
github.com/ulikunitz/xz v0.5.10 // indirect
|
|
go4.org v0.0.0-20200411211856-f5505b9728dd // indirect
|
|
golang.org/x/exp v0.0.0-20240808152545-0cdaa3abc0fa // indirect
|
|
golang.org/x/sys v0.24.0 // indirect
|
|
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 // indirect
|
|
modernc.org/libc v1.55.3 // indirect
|
|
modernc.org/mathutil v1.6.0 // indirect
|
|
modernc.org/memory v1.8.0 // indirect
|
|
modernc.org/strutil v1.2.0 // indirect
|
|
modernc.org/token v1.1.0 // indirect
|
|
)
|
|
|
|
replace golang.org/x/text v0.17.0 => github.com/lordwelch/text v0.0.0-20240505231825-4893f344170f
|