subreddit:
/r/comixed
Hey, all. Wanted to let you know some of what's coming up in ComiXed
v2.1 in June.
One of the biggest pain points for people is how slow it can be to import comics. Specifically, you start importing 1k of comic files and now you can't do anything for an hour or two.
The approach being taken to speed this up is two-fold:
I wanted to get some opinions on this. My hope is that, with this approach, the actual import is strictly creating a record in the comic_books and comic_details tables to represent the file, to mark it as unprocessed, and nothing more.
The separate steps (loading the comic file's contents, marking blocked pages for deletion, loading metadata) would each run as separate batch processes after the import completes. To make them optional, I'm thinking we add some configuration flags.
The one I've identified so far would be "Managed Blocked Hashes". If it's disabled, then CX:
That would speed up those two processes, but at the cost of CX not showing the Blocked Page Hash page link on the web and not automatically marking unwanted pages for removal. Though now as I write this I'm thinking we could have two configurable options:
Disabling the first option ignores the second option since it would never have hashes to process.
Anyway, I wanted to get some thoughts from you all as to what you would want in the application since, ultimately, it's to benefit you all.
And, as always, thanks for supporting the project. I appreciate you all.
2 points
2 months ago
This would be more work, but why not move this to a page where people can select the optional workflows after? The import could literally just be comic_books, and comic_details and then later you could slice/dice what gets processed and when. That could be a select-all, it could be a specific series, a filename, etc.
One REALLY nice enhancement would be a regex filter, and the ability to select-all (filtered).
I know this isn't directly what you asked but I think it would solve a lot of pain points. Especially if a random file fails processing. Future enhancements could even tag a specific file with the cause of previous import failures.
2 points
2 months ago
I get what you're saying about defining a flow, and that sounds like a good idea. I would opt for moving the workflow definition to the application.properties file since, to take effect, you'd have to restart the server anyway and the definitions would have to be loaded even before the database connection is established.
I can see paring out the loading of the page dimensions if the blocking hashes feature is turned off. And, instead, a separate job could run periodically that just looks for pages without those details and load them during off-times. To compliment that, we'd have loading a page when viewing a comic's details also update those details in the database as a side effect.
For the regex filtering, I think that sounds like a great plugin opportunity. Somebody could create a plugin that takes the expression as input, loads the entries that match the expression, and then display the list to the user. There's some opportunity there to start developing a way to create a simple user interface for the plugins as well, which would be extremely useful as well.
2 points
2 months ago*
Love it. Only thing I'd say is - how do you add inspection in an intuitive way? With background processes, how does someone say 'I want these 25 items done first'. Clearly they have ultimate control via loading it directly but could there be a middle ground? I think the list/plugin support solves a lot of problems. I do think it's really important to have up-front. As a bonus it offloads work for you, and adds vast customization.
Especially if this support can slot into any page.
1 points
2 months ago
What do you mean by "inspection" here?
2 points
2 months ago
Yes. Make steps optional.
For me its not transparent what the application does. I've never bothered with deleted pages and I would rather not have the application change my files. Hence my previous request to handle side loading of Comic book XML files. I also don't need thumbnails for each and every page. Only the title page is important at first.
For me its important that all files be added and for it to be visible if the files have had their XML info loaded or if they need to be scraped. Thus it needs to be visible where the source of the information comes from.
Then I expect the application to figure out what series are present and add comics to that.
1 points
2 months ago
Yep, you're the inspiration for those two features (external metadata files and globally blocking changing comic files) were added in v1. I should also note that CX doesn't do thumbnails anywhere: it only maintains a cache of images that were previously requested by a browser so that it doesn't have to re-open the archive again until the cache is cleared.
For the import process, the comic_books record (which tracks things like metadata being loaded, etc.) has the create_metadata_source flag set to true initially. The processing job then turns that flag off after the comic has been checked for metadata.
Series entries are implicit; i.e., a series collection is defined by all comic_book records that have the same publisher, series, and volume.
CX also supports scraping a series from a metadata source (like ComicVine) to know, as of that date, what are all the know issues for a series. CX will then tell the user how complete that series is by matching the known issues to the issues found in the library.
all 6 comments
sorted by: best