the media viewer and thumbnail _right-click->manage_ menus now have a _viewing stats->clear_ action, which does a straight-up delete of all viewing stats record for the selected files. 'edit' will be added to this menu in future
extended the tag autocomplete options with a checkbox to allow 'namespace:' to match all tags, without the explicit asterisk
tag autocomplete options now permit namespace searches if the 'search namespaces into full tags' option is set
the tag autocomplete options panel now disables and checks the namespace checkboxes when one option overrules another
cleaned up some tag search logic to recognise and deal with 'namespace:' as a query
added some more unit tests for tag autocomplete options
the html and json parsing formulae now support negative indexing, to select the nth last item from a list
extended the '1 -> "1st"' ordinal string conversion code to deal with negative indices
the 'hide tag' taglist menu actions are now wrapped in yes/no dialogs
reduced the activation-to-click-accept time that the shortcuts handler uses to ignore activating clicks from 100ms to 17ms
clicking the media viewer's top hover window's zoom buttons now forces the 'media viewer center' zoom centerpoint, so if you have the mouse centerpoint set, it won't zoom around the button where you are clicking!
added a simple 8chan.moe watcher to the defaults, all users will get it on update
the default bandwidth rules for download pages, subs, and watchers are now more liberal. only new users will get these. various improvements to db and ui update pipeline mean the enforced breaks are less needed
when a manage tags dialog moves to another media, if it has a 'recent tags' suggestion list with a selection, the selection now resets to the top item in the list
the mpv player now tracks when a video is fully loaded and only reports seek bar info and allows seeks when this is so (this should fix some seekbar errors on broken/slow-loading vids)
added 'undelete_file' to media shortcut commands
file delete and undelete are no longer hardcoded in the media viewer and media thumbnail grid. these actions are now handled entirely in the media shortcut set, and added to all clients by default (this defaults to (shift +) delete key, and also backspace on macos, so likely no changes)
ctrl+mouse wheel is no longer hardcoded to zoom in the media browser. these actions are now handled entirely in the 'all' media viewer shortcut set (this defaults to ctrl+wheel or +/-, so likely no changes)
deleted some old shortcut processing code
tightened up some update timers to better halt work while the client is minimised to system tray. this _may_ improve some users' restore hanging issues
as Qt is happier than wx about making pages on a non-visible client, subscriptions and various url import operations are now permitted to create pages while the client is minimised to taskbar or system tray. if this applies to your situation, please let me know how you get on here, as this may relieve some restore hanging as the pending new-file jobs are no longer queued up
.
fixes:
clicks on hover window greyspace should no longer propagate up to the media viewer. this was causing weird archive/delete filter actions
mouse scroll on hover window taglist should no longer propagate up to the media viewer when the taglist has no more to scroll in that direction
fixed an issue that meant preview windows were initialising about twenty pixels too short for the first page loaded in a session, and also pages created within nested page of pages. also cleaned up some logic for unusual situations like hidden preview windows. one more cycle of closing and reopening the client will fix the option value here
cleaned and unified some page sash setting code, also improving the 'hide preview window' option reliability for advanced actions
fixed a bug that meant file viewtime was still being recorded on the duplicate filter when the special exception option was off
reduced some file viewtime manager overhead
fixed an issue with database repair code when local_tags_cache is missing
fixed an issue updating a very old db not recognising that local_tags_cache does not yet exist for proper reason and then trying to repair it before update code runs
fixed the annoying issue introduced in the recent string match overhaul where a 'fixed character' string match edit panel would not want to ok if the (now hidden) example string input did not have the same fixed char data. it now validates no matter what is in the hidden input
potentially important parsing fix: JSON parsing, when set to get strings, no longer converts a 'null' value to 'None'
the JSON parsing formula now allows you to select the nth indexed item of an Object (a JSON key->value dictionary). due to technical limitations, it alphabetises the keys, not selecting them as-is in the JSON itself
images that do not load in PIL no longer cause mime exceptions if they are run through the decompression bomb check
.
misc:
boosted the values of the decompression bomb check anyway, to reduce false positives. it generally now has a problem with images with a bmp > 1GB memory
by default, new file import options now start with decompression bombs allowed. this option is being reduced to a stopgap for users with less memory
'MimeException' is renamed to 'UnsupportedFileException'
added 'DamagedOrUnusualFileException' to handle normally supported files that cannot be parsed or loaded
'SizeException' is split into 'TagSizeException' and 'FileSizeException'
improved some file exception inheritance
removed the 'experimental' label from sub-gallery page url type in parsing system
there are several new options for tag autocomplete under the newly renamed _services->tag display and search_:
for 'manage tags'-style 'write' autocompletes, you can now set which file service and tag service each tag service page's autocomplete starts with (e.g. some users have wanted to say 'start my "my tags" service looking at "all known files" and "ptr"' to get more suggestions for "my tags" typing). the default is 'all known files' and the same tag service
the old blanket 'show "all known files" in write autocompletes' option under _options->tags_ is removed
you now can enable the following potentially very slow and expensive searches on a per-tag-domain basis:
- you can permit namespace-autocompleting searches, so 'ser' also matches 'ser*:*', i.e. 'series:metroid' and every other series tag
- you can permit 'namespace:*', fetching all tags for a namespace
- you can permit '*', fetching all tags （╬ಠ益ಠ)
'*' and 'namespace:*' wildcard searches are now significantly faster on smaller specific tag domains (i.e. not "all known tags")
short explicit wildcard searches like "s*" now fire off that actual search, regardless of the 'exact match' character threshold
queries in the form "*:xxx" are now replaced with "xxx" in logic and display
improved the reliability of various search text definition logic to account for wildcard situations properly when doing quick-enter tag broadcast and so on
fixed up autocomplete db search code for wildcard namespaces with "*" subtags
simplified some autocomplete database search code
.
string processing:
the new string processor is now live. all parsing formulae now use a string processor instead of the string match/transformer pair, with existing matches and transformers that do work being integrated into the new processor
thus, all formulae parsing now supports the new string splitter object, which allows you to split '1,2,3' into ['1','2','3']
all formulae panels now have the combined 'string processing' button, which launches a new edit panel and will grow in height to list all current processing steps
the stringmatch panel now hides its controls when they are not relevent to the current match type. also, setting fixed match type (or, typically, mouse-scrolling past it), no longer resets min/max/example fields)
the string conversion step edit panel now clearly separates the controls vs the test results
improved button and summary labelling for string tools across the program
some differences in labelling between string 'conversion' and 'transformation' are unified to 'conversion' across the program
moved the test data used in parsing edit panels to its own object, and updated some of the handling to support passing up of multiple example texts
the separation formula of a subsidiary page parser now loads with current test data
the string processing panel loads with the current test data, and passes the first example string of the appropriate processing step to its sub-panels. this will be expanded in future to multiple example testing for each panel, and subsequently for note parsing, multiline testing
added safety code and unit tests to test string processing for hex/base64 bytes outcomes. as a reminder, I expect to eliminate the bytes issue in future and just eat hashes as hex
cleaned up a variety of string processing code
misc improvements to string processing controls
.
the rest:
double-clicking a page tab now opens up the rename dialog
system:time imported now has quick buttons for 'since 1/7/30 days ago'
all hydrus downloaders now accept percent-encoded characters in the query field, so if you are on a site that has tags with spaces, you can now enter a query like "simple%20background red%20hair" to get the input you want. you can also generally now paste encoded queries from your address bar into hydrus and they should work, with the only proviso being "%25", which is "%", when all bets are off
duplicates shut down work (both tree rebalancing and dupe searching) now quickly obeys the 'cancel shutdown work' splash button
fixed a signal cleanup bug that meant some media windows in the preview viewer were hanging on to and multiplying a 'launch media' signal and a shortcut handler, which meant double-clicking on the preview viewer successively on a page would result in multiple media window launches
fixed an issue opening the manage parsers dialog for users with certain unusual parsers
fixed the 'hide the preview window' setting for the new page layout method
updated the default gelbooru gallery page parser to fix gelb gallery parsing
updated the newgrounds parser to the latest on the github. it should support static image art now
if automatic vacuum is disabled in the client, forced vacuum is no longer prohibited
updated cloudscraper for all builds to 1.2.38
.
boring code cleanup:
all final mouse event processing hackey is removed from the media viewers, and the shortcut system is now fully responsible. left click (now with no or any modifier) is still hardcoded to do drag but does not interfere with other mapped left-click actions
the duplicates filter no longer hardcodes mouse wheel to navigate--whatever is set for the normal browser, it now obeys
cleaned up some mouse move tracking code
clicking to focus an unfocused media viewer window will now not trigger the associated click action, so you can now click on archive/delete filters without moving on!
the red/green on/off buttons on the autocomplete dropdown are updated from the old wx pubsub to Qt signalling
updated wx hacks to proper Qt event processing for splash window, mouse move events in the media viewer and the animation scanbar
cleaned up how some event filtering and other processing propagates in the media viewer
deleted some old unused mouse show/hide media viewer code
did some more python imports cleanup
cleaned up some unit test selection code
refactored the media code to a new directory module
refactored the media result and media result cache code to their own files
refactored some qt colour functions from core to gui module
added 'system:has/has no note with name xxx' to search for specific note names
in the normal system predicate list, the notes pred is now the generic 'system:notes' to launch a combined dialog for both num notes and named notes
favourite tag suggestions are now sorted in manage tags dialog according to the default tag sort
page names will now middle...elide when there are too many to fit into a row (and normally left/right buttons would be added). if the elided tabs still do not fit, the buttons will pop up as before. added a checkbox to options->gui pages to turn this text eliding off
pulled the 'page name' options on that panel into their own box and added some text regarding the 'my big row of import page tabs keeps scrolling weird' issue
when files are pixel duplicates, the filesize and age comparison statements will now have 0 score and thus be coloured neutral blue
the standard text entry dialog now always selects any default text it starts with, so you can now type to immediately overwrite. see how you like it and if there are some places where you think an exception should be made
updated the IPFS interface to work with the new IPFS 5.0. all api requests are now POST so it doesn't 405, and the User-Agent is overridden to one that IPFS will not 403 at, and I fixed a typo the new api is more strict about
a hack to get page splitters to lay out correctly on session load is rewritten from a hammer to a scalpel. pages now set their splitter positions on their first individual visible selection. this both reduces some minor ui lag on session/page load and improves splitter positions for clients that open minimised to the system tray
a long-time odd issue where loaded sessions would initially select the top-left-most non-page of pages is fixed. now the bottom-left-most page of any kind is selected
fixed tag autocomplete selecting the bottom-most pre-loading result. it now correctly selects at the top
fixed an issue setting certain values (typically loading a default) to a tag import options panel
the client is now more aggressive about clearing subscriptions from memory when they are finished running
in windows, the main method that copies files now checks for modified time of the source file. if it is before 1980-01-01 UTC, it does not copy the file metadata, as some Windows has trouble with this lmaoooo
cleaned up how some thumbnail 'current focus' media determination code works. should have fixed some weird errors when hitting certain shortcuts on collections
cleaned up basic list/sort code across the program
the 'queue' and add/edit/delete listboxes now emit change signals when new items are added or imported
pyparsing, a helper for cloudscraper, is now correctly bundled in the built releases. a new line in help->about displays this
help->about now lists cloudscraper version
updated the discord link to the new https://discord.gg/wPHPCUZ
.
upcoming string processing changes for advanced users:
I extended string parsing code this week, but I am not yet ready to turn it on. when it does come on, it will change all formulae from the fixed string match/converter pair a combined general string processing 'script' of n steps
wrote a new 'string splitter' object that takes one strings and splits it into up to n strings based on a separator phrase (such as ' ,')
wrote an edit panel for string splitters
wrote a new 'string processor' object that holds n ordered string match/converter/splitter objects and filters/converts/splits x strings into y strings based on those steps
wrote an edit panel for string processors. it has a notebook that live updates with test results for each step on every update
wrote unit tests for string match
wrote unit tests for string converter
wrote unit tests for string splitter
wrote unit tests for string processor
refactored string conversion edit panels to their own file
refactored string conversion controls to their own file
misc string processing cleanup and labelling improvements
.
technical url parsing stuff:
urls are now stripped of leading and trailing whitespace during normalisation, just in case a paste contains some extra whitespace. previously, it would sometimes throw a 'doesn't start with http' error
the hydrus url normalisation process now normalises the hostname according to the NKFC unicode format, meaning unusual characters like ？and e◌́ are now replaced with their normalised visual equivalent ? and é, and hence these urls will no longer throw errors when they are added
if '?' or '#' end up in a hostname (which are invalid characters), it is now converted to _, just to stop complete parse mangling when weird urls are submitted. this character replacement may become more sophisticated in future
the hydrus downloader should now support search terms that include '#'
download query parameters that contain '%23' ('#', encoded) are now not unquoted in url normalisation
the file notes system is more mature. files now store multiple named notes
the edit notes ui is now a tabbed window with add/edit_name/delete buttons
media results now load with their notes, so note access is instant
thumbnails now show a notes icon when they have notes
the media viewer top-right area shows a notes icon when the current file has notes
clicking the media viewer top-right notes icon opens edit notes
the edit notes menu entry now lists the number of current notes if there are notes
added a 'system:number of notes' predicate. it has easy 'has/no notes' buttons for quick filtering
the file notes database table will be updated on update, it shouldn't take long. existing notes will get the default 'notes' name
duplicate notes now share the same storage space in the database
in prep for a future search expansion, notes are now cached in the database for fast text search
in prep for note parsing, wrote a 'note import options' object. it doesn't do anything in the program yet, but it supports multiple note conflict resolutions, note extension detection, and global and specific note renaming
wrote unit tests for the new note import options
.
some tag search stuff:
hydrus now maintains an internal mapping of direct 'searchable' versions of tags to the tags themselves, which allows it to now do fast exact-match (short search) and complicated wildcard lookups of tags with unusual characters. 'f' and '/f/' will now return '/f/' and 'board:/f/' quickly, 'board:f' and 'board:/f/' will return 'board:/f/' quickly, and 'te*a*' will correctly return 'test-tag'
it will take a few minutes to regenerate this new cache on update
complex wildcards like 's*m*' are now treated the same as simple ones like 'sam*' and should match unusual subtag characters in all cases
wildcard tag file search predicates are now plugged into the new cache, so the search preds '/f/*', 'board:/f/', 'board:/f/ast;', 'b*d:/f/' and 'b*d:/f/*' now all match files with 'board:/f/', as do wildcards that include replacement characters, so the same should be true above for 'f' instead of /f/' in all cases
new wildcard search preds do not collapse their characters for their presentation string, so 'date:2*-01-01' now renders like that, not 'date:2* 01 01'
wildcard file search predicates are now faster for simple (just an asterisk on the end) subtag wildcards
the fts search cache is moved from 'master' to 'caches' db this week, it will take a few moments on update
the 'repopulate tag search cache' db regen job now repopulates the fts cache, the new 'searchable' cache, and the integer tag cache
the database repair code now checks for the fts cache and new 'searchable' cache on boot and, if they are missing, warns the user and creates empty tables
.
improvements:
fixed the unsorted tags in tag suggestion boxes
clicking the inbox icon in the top-right hover window now archives the file
system:dimensions now has quick buttons for 16:9, 9:16, 4:3, 1:1, 1080p, 720p, and 4k
system:known url searches are now better about fetching www and non-www urls for the domain or url class
the edit shortcut sets panel now has nicer english names for reserved shortcut sets, and also sorts them in a more logical way
you no longer have to be in advanced mode to copy file hashes from thumbnails
users in advanced mode can copy the internal file_id of files from the thumbnail/viewer copy menus (this is most useful for the client api)
system num_frames, num_words, and num_notes now display alternate 'has/no xxx' labels when they search for =0 or >0
you can now search for 0 with system:num_frames
.
fixes:
users who could restore from system tray using the menu but had trouble with clicking _should_ now have better luck with clicking
fixed some instances where fps could be calculated as 0, which would lead to other problems down the line. now a missing or 0 fps is remapped to 1
fixed system:framerate for '<' queries
the status bar cells now get expanded tooltips to describe what they do
fixed some media result caching code that could in rare cases cause an error in content update processing when the result disappeared from the cache during processing
the 'hard-replace siblings and parents' button on 'manage tags' now makes a submenu so its actions' long labels show better
fixed a handful of tables that were not starting sorted
a variety of credential parse and other server failures that were formerly returning 403 now properly return 400 and 409
in order to improve default 'open externally' behaviour on Linux/macOS, if the environment variable XDG_DATA_DIRS is not preserved through a hydrus build launch env, hydrus now sets a simple 'default' value for this before running xdg-open
if the client is booted from a windows shortcut to a built release, the program restart command is slightly more reliable
.
misc:
cleaned up some db update error reporting code, it should now more reliably make an english-friendly popup text box before splurging technical info
refactored some media object code, cleaned some class definitions, and added typing hints
misc code cleanup
the 'getting started' help files now have anchor definitions, so their sections can now be #linked to
added several links in the 'getting started' help to the user-created video guides here: https://github.com/CuddleBear92/Hydrus-guides thank you for making these!
added a link to the help for the user-made 'other archiving software' guide here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/wiki/0-Alternative-Programs-and-Resources#software thank you for making this!
fixed link to AUR package in the help
updated cloudscraper in all builds to 1.2.36
updated windows mpv to a significantly newer dll, it now reports api version 1.108
included libgpg-error.so.0 in Linux build, which will improve some Linux situations (more reports from Ubuntu 20.04 or others about missing/conflicting .so files are welcome)
favourite tags now correctly refreshes on new media
the tag suggestion lists in manage tags now discard current and pending tags that _all_ the current media already have, and all tag suggestion lists update this filter any time the media gets a tag content update! they _should_ update live now
all tag and predicate taglists now try to move the selection to a 'nice' neigbour when a keyboard enter activation results in the current selection being removed (e.g. as in these tag suggestion lists). the nice selection should be the tag after, before, or at the top of the list, and should make it nicer to keep navigating the list and add tags with your keyboard
all tag and predicate taglists now try to preserve selection on simple clear-and-set data refreshes
.
deleted tags overwrite update:
due to an unfortunate oversight, until now tag parsing has not filtered out previously deleted tags from the tags it parses and sends to the local database
as the majority of downloaded files are parsed once per site per user and in a similar time window before manual editing ever occurs, and most non-tag-sibling-eligible bad tags are site specific or not parsed to begin with, and as these undesired tags were not broadcast up to the tag repository, this problem has not been very obvious and I believe has not affected most users too much. this is however a reason why some users who have more recently downloaded many older files are seeing smaller 'deleted mappings' counts on their ptr review panel (and some low quality tags in their db), as they have been re-adding previously deleted tags to their local store
this has been fixed. tag import options now load the pending importee file's metadata before tags are filtered and discard currently deleted tags from those to be added or pended. this applies to parsed tags, additional tags, and those tags added through special other means, such as from a parent gallery page.
if you do wish to allow parsed or additional tags to overwrite currently deleted tags for a particular job, the cog icons on the edit tag import options panel now allow you to permit overwrite for either
tags added via hard drive imports or the migrate tags tool still overwrite deleted tags as before
as this is a local-only problem, there is thankfully a retroactive fix for this issue for tag repository domains, involving a content reprocess run to re-apply deleted tags. I am not activating this automatically this week as this is a heavy job for the ptr and I need to study the true fallout of the problem more, but I may in future, likely as a smaller and more targeted maintenance job. advanced users can do it now under the ptr's review services panel
I regret missing this, and I am sorry for any inconvenience. I only discovered it through the serendipity of some users recently reporting unusual deleted counts and a personal item in my todo to check the reliability of deleted mapping filtering for local tag domains--turns out it never got added, and we never specifically noticed, fugg
there are now unit tests for the improved tag filtering pipeline and both of these new overwrite options
.
the rest:
hydrus can now use several different zoom 'centerpoints' about which to expand and shrink a zooming file. this was previously hardcoded to the center of the media. under options->media, you can now set it to be the media window center (the new default, which feels much nicer after a pan), the mouse cursor, the old media center, or the media top-left corner
cleaned up the related zoom positioning code, and removed the jarring old re-centering off-screen rescue hack when zooming out to canvas zoom
added a warning about big zooms to the media options page
fixed tag autocomplete filtering in python 3.7 so 'character:aran' matches 'character:samus aran' again
when the hover windows on a media viewer have focus, they _should_ now pass up all options->shortcuts shortcuts to the media viewer
mouse back/forward buttons _should_ now be supported in the shortcuts system, as much as your OS allows them to work like regular clicks
fixed a rare crash with the 'clear trash' button
the client will now not re-analyze tables that have been previously scanned with at least 100k rows in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit
the client will now not vacuum database files greater than 1GB in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit
the new 'cannot vacuum because xxxx' log entry is now only ever printed once per boot. however due to the above change, it likely won't appear in the normal maintenance cycle anyway now
cleaned up some vacuum code
reworked the panel system to better test data validity vs 'woah, you sure you want to do this?' tests and generally cleaned and simplified the canok/cancancel/isvalid testing logic for all panels. panels like manage siblings will now not produce two message boxes if you try to ok them on an uncommited pair and then back out of the ok
refactored the top level window code and improved scrollable panel code typing
more standalone gui function code refactoring
fixed a click-selection-test bug when clicking on certain whitespace in certain predicate lists
the text of the cloudflare-specific error when encountering a captcha page is improved
cleaned up some tag list menu copy and select code, both the menu labels and the copy action, for unusual tags. the 'copyable tags' fetching code is now flexible and unified for menu and action
cleaned up the taglist sibling copy code, eliminating the chance of dupes
fixed a _little_ of the wording on the discard/exclude tag list menu labels for negated predicates, it still feels a bit awkward and I will keep working here
cleaned up some old media metadata fetching code
misc import code typing
misc list/iterable typing improvements
added some misc media-tag tool code
unified the tag import options tag filtering pipeline somewhat to deal with the deleted overwrite situation
improved a debug ui test to no longer need window focus
the text you type into tag autocomplete is now parsed in a unified object. all the variants of empty text, invalid text, valid text, namespace text, and wildcard text are all tested and fetched in one simple location with better code
autocomplete results caching is now a unified object that tracks and filters results in one location. wildcard searches are now never cached by accident, and switching from tag cache to system predicate cache and to non-initialised cache is instant and more reliable
when an autocomplete, either in a search page or a context that manages tags, has results include multiple sibling variants of the typed text, they are now all elevated to the top of the list. the ideal is at the top, the entered text is next, and any known siblings follow
the search character 'collapse' that ensures quote marks and hyphens and other odd characters are unified across tags now applies uniformly to all non-complicated-wildcard search tags, with namespace not collapsed and subtag always collapsed
when entering an explicit wildcard search, both strict and autocomplete versions (whether they end with an asterisk) are now displayed
the way tag results are filtered is now more accurate for some unusual wildcards
it is now more difficult to slip cpu-killer search tags (weird asterisk combinations) through
the quick-broadcast that happens when the user hits enter before any results have started loading now uses the unified object and chooses a safer and more reliable broadcast value. the test whether to do the quick-broadcast is also more reliable, particularly in unusual situations where a recent search was cancelled or delayed. note that for many users, the cache and search tech is fast enough that this very rarely triggers
searching with a wildcard below the autocomplete threshold can no longer trigger a full search, nor an invalid exact-text search
namespace count merging is now unified across db tag fetches and media fetches
include current/pending buttons now filter down to media-based tag autocomplete counts
namespace tag autocomplete queries will no longer show up some unusual siblings below the 'anything' tag
deleted a whole bunch of old a/c and caching code
added comprehensive unit tests for the new parsed autocomplete text object
added comprehensive unit tests for the new predicate results cache object
.
the rest:
fixed a stupid typo bug in the new domain checking code that was stopping subscriptions with incomplete file queues from starting. I apologise for this
network error responses 502 (Bad Gateway) and 503 (Service Unavailable) are now treated as a retryable. the 503 is assuming it is not a CF challenge page. if they fail all retries, they are considered a network infrastructure error
all other misc 5xx http responses are now treated as instant network infrastructure errors and will be logged in the new domain health tracker
the exit splash screen now opens a bit earlier, so you now shouldn't have any momentary uncertainty where no windows are open
clients that start minimised to system tray _should_ be better about restoring splitter positions on first show
the various 'management panels', the panels on the left of main gui pages, now have smaller minimum width where available. the gallery and watcher panels are still the widest, which is a limitation of the current list tech. when it gets better column sizing code and selection memory, this will improve
fixed an issue loading gifs with some OpenCV versions
brushed up some running from source help
deleted the Py2To3 script that attempts to detect a legacy python 2 install
improved all the gui files' import order
cleaned up and refactored some subscription code
added a bunch of type hints to edit panel code
misc code cleanup
.
environment updates:
did second step of hydrus project structure improvement--now the project is split into subdirectories for core/client/server/misc and some client subdirs. work here will continue
linux build gets some new libraries, cv is up to 4.2.0
it isn't important, but hydrus is now built in python rather than directly from command line. my build scripts now include cloudscraper and the new hydrus source code tree in the build as they are, rather than hardcoded copying
the hydrus client now has an experimental hook to the cloudscraper module, which is now an optional pip module for source users and included in all built releases. if a CF challenge page is downloaded, hydrus attempts to detect and solve it with cloudscraper and save the CF cookies back to the session before reattempting the request. all feedback on this working/breaking irl would be welcome. current expectation for this prototype is it can pass the basic 'wait five seconds' javascript challenge, but only a handful of the more complicated captcha ones
if a CF challenge page is not solvable, the respective fail reason for that URL will be labelled appropriately about CloudFlare and have more technical information
.
the hydrus network engine now has the capability to remember recent serious network infrastructure errors (no connection, unsolvable cloudflare problem, etc..) on a per domain basis. if many serious errors have happened on a domain, new jobs will now wait until they are clear. this defaults to three or more such errors in the past ten minutes, and is configurable (and disableable) under options->connection. this will be built out to a flexible system in future, with per-domain options+status ui to see what's going on and actions to scrub delays
basically, if a server or your internet connection goes down, hydrus now throttles down to limit the damage
subscriptions now test if a domain is ok in order to decide whether they can start or continue file work, just like with bandwidth
serverside bandwidth alerts (429 or 509) are now classified as network infrastructure errors
I expect this system will need more tuning
.
the hydrus downloader system now recognises when an expected parseable document is actually an importable file. when this is true, the file is imported. this hopefully solves the situation where a site may deliver a post url or a file
.
the rest:
the windows build of hydrus is now in python 3.7.6, up from 3.6. this rolls in a host of small improvements, including to network stability and security (e.g. TLS 1.3), and possibly a couple of new bugs in more unusual hydrus systems
similarly, all the windows libraries are now their latest versions. opencv is now 4.2
greatly sped up several file searches that include no tags such as bare system:rating, most system file metadata predicates, or bare system:inbox, when the result size is much smaller than the total number of files in the file domain
thanks to some excellent work by a user, the Deviant Art downloader gets another pass--it can now get high res versions of images where they are available, and video, and flash, and pdf! the only proviso is that you need to be logged in to DA to get most content, otherwise you get 404. the current hydrus DA login script _seems_ to work ok
tag import options blacklists now test unnamespaced rules against namespaced tags. so if you blacklist 'metroid', a 'series:metroid' will be caught and the blacklist veto signal sent. this can be escaped with the 'advanced' exception panel, which now permits you to add 'redundant' rules
the edit tag filter panel now explains the blacklist rules explicitly and has a second 'test' green/red text to display test results for a tag import options blacklist, with the new sibling and namespace check
added some unit tests to test the new tag import options blacklist namespace rule
when 'default' tag import options are set, the edit panel now hides the per-service options, rather the the previous disable
the system tray icon now destroys itself when no longer needed, rather than hiding itself. it should now be more reliable in OSes that do not support system tray icon hide/show. if your OS still doesn't get rid of them, and you get a whole row of them, I recommend just leaving it always on
the system tray now has a tooltip with the main hydrus title and pause statuses
the timer that hides the mouse on the media viewer is now fired off when the window first opens (previously it would only initiate on the first mouse move over the window), so users who navigate mostly by keyboard should now see their cursors nicely hide on their own
added some semi-hacky import/export/duplicate buttons to edit shortcuts. I'll keep working on this, it'd be nice to have import/export for whole shortcut sets
added a semi-hacky duplicate button to the 'manage http headers' dialog
the 'clear' recent tag suggestions button is now wrapped in a yes/no dialog
a new checkbox under options->gui now lets you set it so when new cookies are sent from the API, or cookies are cleared, a popup message summarises the change. the popup dismisses itself after five seconds
the client api now also returns 'ext' on /get_files/file_metadata calls, just as a simpler alternative if the 'mime' is a pain
fixed a bug when petitioning tags through the client api, with or without reasons
fixed an error where subscriptions that somehow held invalid URLs would not be able to predict some bandwidth stuff, which would not allow the edit subs dialog to open
the string transformation dialog's step subdialog is now ok with example strings that are bytes. even then, this str/bytes dichotomy is an old artifact of python 2 and I will likely clean it up sometime so string transformers (and downloaders) only ever work utf-8 and hashes just work off utf-8 hex
added a BUGFIX checkbox to options->gui that tells the UI to use Qt file/directory picker dialogs, instead of the native OS one. users who have crashes on file selection are encouraged to try this out
updated running from source help with cloudscraper, a new pip masterline, and some windows venv info
the 'import with tags' button on 'import files' dialog gets another rename for new users, this time to 'add tags before the import >>'. it also gets a tooltip
handled an unusual rare error that could occur when switching out a media player inside a media viewer, perhaps during media viewer shutdown
the hydrus client db now maintains a fast cache of current+pending tag-to-ideal-tag sibling relationships. it works for specific services and 'all known tags'. this is a nice tool and the first step in having a proper hard-baked siblings mappings cache
the new sibling cache can be regenerated under _database->regenerate_. the 'autocomplete cache' entry under that menu is also renamed to the now more appropriate 'tag mappings cache'
the db repair system can regenerate this new cache if any part is missing on boot
the lookup that finds tag sibling matches for autocomplete uses this and is now faster, specific to the searched service, more accurate about status, and now includes pending siblings
wrote a new unified object to manage a collection of tag siblings, it is now in use at the db level
as I continue to develop this new fast tech, the old 'apply all sibs to all services' option, which was always buggy, may sometimes not apply in it. I will ultimately replace it with a fuller per-service choice system that will work quickly and properly and in the same unified way
fixed a bug where only one local tag service's siblings would be matched at the ui level when looking at 'all known tags'
fixed a bug in the file search code where searching for a tag that had an unnamespaced sibling going to it would result in searching for all possible namespaces of that sibling (e.g. searching for 'character:samus aran' when 'samus aran'->'character:samus aran' sibling existed would result in effectively 'anything:samus aran')
when tag services are deleted, they are now better about cleaning up their siblings and parents quickly
optimised some tag and hash id->value database cache population routines to improve performance for large queries (e.g. when fetching all the tag parents/siblings on boot). also these caches are now larger, 100k instead of 25k
all cache regen code now forces an immediate analyze of the new tables to speed up imminent access
.
the rest:
updated the default e621 file page parser to get rating tags again (looks like their markup just changed again)
updated the default sankaku file page parser to get their recently redefined 'genre' tags
in edit subscriptions, the 'overwrite tag import options/check options' actions now initialise their dialogs with the current value for the first subscription, rather than the global program default
in the edit subscription panel, the checker options button is moved down to the file/tag import options
when not in advanced mode, the edit tag import options panel now has some red-text at the top to reinforce to new users that they should generally use the defaults
the tag import options blacklist now secondarily checks against all known siblings of the parsed tags, rather than just the 'collapsed' ideal siblings
subscriptions are now more aggressive about clearing out old urls from their file import caches--instead of clearing the 251st url after it has aged twice the death period, now they use just one DP. also, checkers with static checker timings will use five times that check period as DP if that is smaller. static checkers, or those that never die, will use a flat value of six months as DP if that is smaller
moved a bunch of the debug 'data actions' to a new 'memory actions' menu
significantly reduced how often the system tray regenerates its menu, which seems to improve stability
fixed an issue where guis that were maximised before a minimise were restoring from a system tray icon click to normal view
double-clicking the system tray when the ui is hidden should no longer do a fast show/hide
fixed an issue where if the gui was minimised, the main animation timer would not run for other windows (e.g. a separate media viewer)
improved ui shown/hidden tracking logic for the new system tray icon for different OSes
fixed the 'refresh_page_of_pages_pages' shortcut action, which had faulty old wx code in it
fixed a wx->Qt bug where modal popups that cannot be cancelled, and thus pop up a 'sorry, you can't dismiss this' text when you try to close them, were nonetheless still closing afterwards
the hydrus client and server now attempt to listen their servers on both IPv4 and IPv6, failing gracefully if IPv6 is not available
the 'is this a localhost request?' check now understands IPv6 localhost (::1 or ::ffff:127.0.0.1)
may have solved a 100% cpu repaint issue with the a/c dropdown in some qt environments
added info to installing help about Windows N and clean installs
misc media viewer wx->Qt code cleanup
misc code cleanup
.
experimental hellzone, be wary ye scabs:
added an experimental 'sub-gallery url' url content type to the parsing and downloading system. this url is queued into the gallery log even if the primary gallery page found no file/post urls, and is meant for galleries that link to galleries. not yet ready for primetime, but feedback would be appreciated
added an experimental ui-hang relief mode, activated under _help->debug->data actions->db ui-hang relief mode_, which _should_ stop the ui hanging in unusual long-time ui-synchronous db jobs. it may cause other problems, so it is default off. it also prints begin/end statements to log for additional info. users who experience ui hang due to db job processing time are invited to play with this mode and report back results
hydrus now can now make a system tray icon for those OSes that support it. it can be buggy/crashy under non-Windows, where it gets some warning labels
under the new options->system tray page, you can set whether to show the system tray all the time, minimise the main gui to system tray, close-button the main gui to system tray, and start the program minimised to the system tray
right-clicking the icon brings up a menu to show/hide the ui, pause/unpause network traffic or subscriptions, and to exit hydrus
the main file menu now has an option to minimise to system tray
double-clicking or middle-clicking the icon will show/hide the whole hydrus ui as long as there are no dialogs open
clicking it will restore the main gui from minimise or raise it to the front
on an ui hide, the current preview window will be blanked and media viewers will be paused, so any ongoing noise/cpu from them should stop
a new 'global' shortcut 'hide_to_system_tray' is now available
starting the client minimised may have some layout issues on first show--I particularly had to fix splitter layouts--please report any more you discover
.
framerate and num frames:
system:framerate search added to system:duration panel. precise framerate is tricky with current hydrus info, so it searches +/- 5% of a given value
system:number of frames added to system:duration panel
sort by number of frames added
duration/framerate/num frames sort moved to their own 'duration' submenu
framerate added to generic media metadata summary string (which appears in status bar and media viewer, etc...). precise framerate is tricky with current hydrus info, so it is rounded to the nearest integer
.
the rest:
rolling in new danbooru file page parsers that should fix file downloads, thank you to a user for the submission
rolling in a e621 login script, thank you to a user for the submission
gave tag autocomplete results fetch code a pass, cleaning up several instances of incorrect or inefficient timing and caching logic and I believe fixing the issue where system preds would sometimes not be loaded after entering a tag
improved reliability of autocomplete dropdown hiding on background pages (some edge cases where these could still hang around _should_ be fixed)
improved 'hide' tests in several parts of the program related to the new system tray icon, which should help some other cases--e.g. weird shutdowns now _shouldn't_ ever leave a bunch of floating popup messages
fixed a bug where pages set to open with all known files/tags domains, which is not supported, was incorrectly substituting tag domain with a file domain, which is even more not supported
cleaned up some sort code--I believe this has fixed the odd issue where a 'time imported' sort would not work on some pages (such as one loaded from a favourite search)
fixed the 'related' tag suggestion box not knowing about new pending tags added in a manage tags dialog open on a media viewer after next/previous media transitions while the dialog is open. also it and the file lookup's lists now clear when a new lookup starts
the tag suggestion boxes are now add-only and remove what you add as you add them! let me know if this feels nice or not!
the splash window now has a different 'booting/exiting' window title, if you would like to hook it with a window manager
went over all the 'prep url for display', 'filter urls', and 'normalise url' requests across the program to deal with invalid url (e.g. garbled text) better
you can now no longer add invalid urls via the client api associate_url call--you'll get 400 instead
cleaned some thumbnail selection and rendering code, particularly fixing some edge case 'where that media go?' issues where collect-by calls happen during thumbnail waterfalls and so on
cleaned up some page file domain setting code and misc page management code
improved accuracy of rendered image cache memory footprint calculations
fixed some Qt signal object definitions that were causing errors for some users who run from source
fixed a bug that was causing potential duplicates to be sometimes re-added between media groups that were previously set as false positive/not related. I apologise for the inconvenience this bug has caused. if you were hit by this, please reset your potential duplicate pairs (hit the cog button on the dupes page) and re-search, and the bad pairs should not be re-added again
fixed an issue where tag autocomplete entry in the form 'namespace-blah:' was replacing the hyphen or other 'collapsable' character to space, which then was not searching correctly for the _anything_ namespace search
'namespace:anything' searches now work when the namespace itself has a wildcard
fixed 'write' autocompletes not matching inputs with UPPERCASE letters
fixed adding tags that start with a colon (e.g. ":D") in 'write' autocompletes
it should now be impossible to enter some 'kill my cpu' queries into tag autocomplete, such as '[asterisk]:anything', even if accidentally entered through the fast-add system
the 'cancel search' stop button that appears after a search takes three seconds is back to being neatly embedded beside the tag autocomplete input box
hitting the cancel search button now clears the non-interactable 'Loading' thumbnail media page (with its misleading 'Loading...' statusbar) and returns you to an empty thumbnail page
loading a favourite search with non-immediate search no longer loads the 'loading...' page. it also saves that new non-immediate status to the page session more reliably
reworked my linux build environment (pyinstaller=3.5, virtualenv=16.1) so that you can launch the built exe using a symlink
rolling out a first version of a requirements.txt, any feedback would be appreciated
rolling out another version of the derpibooru file page parser that no longer duplicates namespaced tags as unnamespaced, thank you to the user who submitted this
.
boring stuff:
.
moving old pubsub system to Qt signals:
all the 'refresh query' calls that do changes to the current file search across the program
the current file and tag domain update calls for search pages
the clear/set file focus calls when launching and exiting the media viewer browser or archive/delete filter
the way thumbnails send the current focused media to the preview media window
the way widgets with shortcuts-based tooltips are notified to update those tooltips when shortcuts change
the way a thumbgrid sends the current tags to be displayed in the 'selection tags' list
the way a thumbgrid adds newly imported files' tags to the 'selection tags' list as they fade in
the way the 'searching/waiting' search button is flipped on and off by shortcut. btw what should be the correct name + label for this button? should it really be an icon?
.
cleanup:
NOTE: the 'include' folder is renamed this week to 'hydrus'. if you have source patches, please update. as I further disentagle code in future, hydrus will ultimately move to typical nested folder/module structure
decoupled the shortcuts edit ui code from the controller and db, unified how shortcuts are edited, and eliminated db wait when booting shortcuts editing
decoupled the shortcuts manager from the controller, cleaned all the code, and moved to a nicer reference with proper typing hints
refactored the frame and media controls of clientguicanvas into separate files
renamed the hoverframes file to canvashoverframes and updated its classnames to 'canvas' rather than the old 'fullscreen'
fixed two wx->Qt typo artifacts in the login script edit ui
reduced some occasional idle memory bloat of clients that have large subscriptions
cleaned up how media-based taglists are appended with new media
the e621 file page parser is updated again, thanks to a user's contribution. this one gets md5 and file url more reliably, and also gets rating tag
added a 'e621 file page (old format)' url class to help match and search for files downloaded with the old format. please be aware there is no good solution to auto-convert old urls to a new format yet, so this connection does not (yet) solve the old/new comparison test
updated deviant art file post parser to use their json api. this should be more resilient to their current layout changes
the nijie.info login script appears no longer to function. as with exhentai last week, it has been removed to make it easier to log in with hydrus companion. please use hydrus companion if you would like to log into nijie.info
updated file lookup scripts for 'iqdb danbooru' and 'danbooru md5' thanks to a user's contribution
.
the rest:
the way the mpv.conf works changes this week. it is now correctly fully portable, stored in the db directory beside the .db files. if this file does not exist, the 'default' as stored under the install_dir/static/mpv-conf folder will auto-populate it. if you have been using a non-default mpv conf, please re-set it one time after update, and you should be good
the code that loads mpv.conf is now more graceful on 'missing file' errors, which now means when both the db conf and the default conf are missing
hitting escape on a tag autocomplete input that has text will now clear that text! note that hitting escape on an _empty_ a/c input will still do 'lose focus' and then 'close dialog'
updated the slideshow logic so that if a media with duration has a shorter duration than the slideshow duration (e.g. a gif that lasts 0.5s on a 10s slideshow), the media will keep looping until the duration is up. media that has duration longer than the slideshow time will continue to play through once completely, delaying slideshow progression and then stopping promptly when it has finished
the string transformation system now allows 'url percent encoding' under the encode/decode type!
fixed the 'only add existing tags' filter in the tag import options, which was denying all the tested tags. it seems to have been hit by a typo in the last three months
the 'favourite searches' defaults now include an 'empty page' entry, which is a convenient way to simply clear a page. all users will also get this on update, feel free to delete if you don't like/need it
opening a new search page from a tag or an active search predicate ('open a new search page for...' or middle-click) now copies the file service (e.g. looking at trash) from the original page
opening a new search page in the 'all known files' file domain when the tag domain should be 'all known tags' (a currently unsupported combination) now coerces the tag domain to 'all local tags'
checkboxes should now appear again on the collect-by dropdown in Fusion (and hopefully any other) style
fixed an issue where entering 'namespace:*' explicitly would show the much less efficient wildcard search rather than the efficient 'anything' namespace search
fixed an issue where wildcard search could include multiple asterisks in a row
fixed an issue with page duplication where the main management object was not being duplicated properly until a session reload, meaning the two pages would sometimes share signals and changes
an old wx delayed hide/show performance hack is removed, making the floating autocomplete dropdown now update more smoothly to resize or move requests, such as when the main gui window is dragged
the program base installation directory is now calculated more accurately, both when running from source and the frozen build, and when launched using a symlink
install dir and db dir are now specified in the help->about window
the petition page content checkbox list now has a taller minimum height
improved error text reporting in hydrus service login failure, hydrus service delay reason-setting, and all 'cancelled' errors across the program
the review services panel now has elided... text. when unusually long errors propagate up to its status texts, it now won't suddenly jump to 2,000 pixels wide. full text appears in tooltips
code refactoring: the tag autocomplete input now now takes responsibility for the active predicate list above it
refactored some tag lists and added typing hints to improve how current page predicates are determined
did some prep work for tag filters supporting wildcards, but it isn't ready yet
hydrus can now save, load, and edit favourite searches. this first system stores searches with a name and an optional folder name, and contains search predicates, file and tag domain, whether the search is live or not, and optionally sort-by and collect-by
this is program-wide and all accessed through the new 'star' icon menu button beside any 'read' tag autocomplete input on search pages, duplicate pages, export folder ui, and file maintenance selection
wrote a favourite searches manager
wrote a dialog to manage favourite searches
wrote a dialog to edit a single favourite search
wrote load and save search functionality
autocomplete dropdowns that have buttons beside them now stretch their floating dropdown windows across the button width also
cleaned a variety of search code, simplifying objects and responsibility
cleaned up some collect-by ui code
refactored sort and collect controls to better location
refactored search constants
numerous small search code fixes and cleanup
renamed clientguipredicates to clientguisearch
.
the rest:
a note from the users managing Hydrus Companion: The Chrome Web Store release of Hydrus Companion is no longer available due to publishing issues. If you have been using it in the past, please install the extension manually as outlined here instead: https://gitgud.io/prkc/hydrus-companion
the default e621 downloader is updated to their new system, thanks to a user's submission. if you log in to e621 with hydrus or the hydrus companion and discover some tags are now blacklisted, please check your blacklist settings on your account on the site
an old test e-hentai login script from 2018 that is no longer in the client defaults will be deleted from clients that still have it today. if the user has no other login script for e-hentai, the domain entry will be deleted as well. this removes potential technical barriers for users that wish to use hydrus companion to access e-hentai, which is now the recommended method
hydrus mpv now has an appropriate stream title, which propagates up to the os-level sound mixer. it was previously the ugly hydrus filename
improved error handling when mpv is passed an invalid conf
the default mpv conf now has audio normalisation that seems to work ok
fixed an issue with the 'delete/move out missing/corrupt file' file maintenance job where record deletes were not processing correctly. it now deletes the file record correctly and also clears that deletion record, to make re-import of the correct file, if found, easier
all hydrus menu labels are now "middle...elided" when they are greater than 64 characters
all new hdd, url, and simple download pages should now obey the 'remove files when trashed' rule. pages in existing sessions will not
updated the user-created CutieDuck darkmode qss file to the latest version, which alters the recent hydrus qss styling colours like green/red button labels
did a full pass of all service fetching--all file and tag services should now present in lists and tabs in service_type, alphabetical order, e.g. for manage tag siblings, the tabs will always be local_tags, tag_repositories, both in alphabetical order
fixed an issue where a 'get darker or lighter comparison colour' calculation was not working well for black or very dark colours
if subscriptions or general network traffic is paused, the bandwidth section of the main gui statusbar now says it
the status bar now tooltips each section
clarified some labels on the edit url class panel
moved all delayed focus-shifting code to a more stable system
cleaned up how the global icon cache is initialised and referenced
updated the hydrus project gitignore to hide all db, log, server, recovery, and media files that could be under the db directory
updated the endchan links in the help to have a .org secondary link
the sort-files-by dropdown is now a button that launches a nested menu. it still supports mouse wheel events. it should now be quicker to find what you want!
added 'sort by framerate' to regular file sort. it works for file search at the db level as well, when mixed with system:limit
under options->sort/collect, the namespace sort-by ui has finally had its makeover. it now has add/edit/delete buttons and up/down buttons for reordering how the entries will appear. it also deals with bad input better. furthermore, namespaces that have hyphens (like 'creator-id') are now supported in namespace sort (and hence collect-by dropdowns!)!
numerical (multi-star) ratings can now be set by dragging the mouse across the line of stars
added 'duplicate page' to the page tab right-click menu! it just makes a copy of the page or page of pages right beside it
system:everything will now always show up in non-query-page autocomplete dropdowns (such as in the file maintenance dialog)
wrote a maintenance routine to repopulate and correct the tag text search cache. it is possible to trigger this (though it is typically pointless) from the database->maintain menu
updated the characters that are ignored in autocomplete tag text search rules, which help skip over unusual characters and assist word-break discovery for searching for tags like '[intensifies]'. as well as the previous brackets, braces, paretheses, quotes, and double-quotes, now slash, backslash, hyphens, and underscores(!) are ignored. searching for 'bbb' will now match a tag 'aaa-bbb', and searching for 'blue_eyes', 'blue-eyes', 'blue eyes', or 'eyes' will match all of 'blue_eyes'. 'blue-eyes', and 'blue eyes'!
to effect the above change, the client will take a few seconds to a minute to update
the above tag text search rules now collapse contiguous unusual characters, or combinations of whitespace and characters, better
namespace and simple wildcard search inputs no longer have the tag text search rules applied to them, meaning you can now search for these unusual characters more specifically when desired
updated the derpibooru gallery search objects to use their api, thanks to a user's submission. this re-enables the 'no filter' mode
added watcher support for tvch.moe, which works with an existing 4chan-style parser
the 'add the ptr' help item now warns the user about the ptr's modern drive storage requirements (4GB download+files, 25GB db). the help files are also updated
I believe I fixed the sometimes crazy fast media drag-move that could happen in archive/delete and duplicate filters
fixed an old uncaught wx->qt issue with the simple downloader where editing the formulae would throw an error
fixed a bug in the 'move highlighted thumbnail' code in the rare case where the currently focused thumbnail can not be found
text input dialogs are now mostly wider
refactored some ui code, cleaning up core objects and import hierarchy
did some controller/gui refactoring, pushing on untangling things
cleaned up a bunch of no-longer-used import statements
misc ui code cleanup
slight rewording of database menu
prepped shortcuts system to ignore a window-activating click (for the media viewer filters), but can't turn it on yet as media viewer clicks are not yet fully plugged in
the client now parses gifs for loop count metadata (some gifs say they should only be played x times through, usually 1). options->media now has a checkbox to control whether this value should be obeyed. both the native viewer and the mpv viewer should follow this. default value is still to loop indefinitely
if gifs are set to play with the mpv player, those without duration will now still be loaded in the native image viewer. the media viewing options ui now notes this
the mpv.conf file used in the mpv window can now be changed under options->media. it _should_ update the conf for all open mpv players on options dialog ok. added to the hydrus static mpv-conf directory are three new 'test' mpv confs for high quality and two audio normalisation tests. all test feedback and recommended conf info is welcome
.
ui cleanup and improvements:
the media viewer mouse autohide time is now customisable under options->media, including disabling it completely. it defaults to 700ms
improved the timing and reliability of the media viewer mouse autohide code
the mouse should now never autohide while a dialog is open
improved the bad colours of the splash screen. it should now be all one colour now, no ugly stand-out white square or other hardcoded colours. hydev also deployed his unparalleled gimp skills to get a white fade around the transparent-background hydrus icon, so it should look correct in darkmodes as well
created a default_hydrus.qss file in the qss folder in order to handle formerly hardcoded colours using hydrus-specific classnames and properties. as well as being loaded by default, this qss file is prepended to any custom stylesheet, so any custom stylesheet that includes its own versions of the hydrus-specific entries will override the defaults. this qss will get more work in future
added on/off buttons to hydrus default qss and converted existing object to use class and properties to obey this
added a variety of valid/invalid/warning text colours to hydrus default qss and converted existing text objects to use classnames to obey this
added accept/cancel buttons to hydrus default qss and converted all green/red buttons across the program to use classnames to obey this
the migrate database dialog now has an outright 'remove location' button to reduce confusion and speed up removal of high weight locations
if a location does not exist on the migrate database dialog, it will now stop throwing multiple error popups every time the list slightly changes, and will complain if file rebalancing is attempted, and will provide different 'remove' yes/no messages if that missing location currently has files or not
slight ui touch-ups to the migrate database dialog
if a window that remembers its position attempts to re-position to a location not on a current display, the windowing system now attempts to rescue it to the primary display, with appropriate popup messages given and errors caught more gracefully
extended these off-screen rescue calls to windows that pull their position from their parent. e.g. if you open the options dialog while the main gui is half over the left side of your screen, it should rescue to the primary display
windows that position off the center of their parent now calculate that reliably on the parent window, not just the parent widget, which never really worked as intended
windows that have no position memory and no parent to pull center/topleft position from will now appear center/topleft of the monitor your mouse is on
the splash screen now appears centered on the monitor your mouse is on
cleaned up and improved a bunch of window/screen coodinate code, moving 'space on screen' calculations to 'space on screen minus taskbar' and similar
unified a 'dialog is open' check across the program
cleaned up the old wx->Qt size, coordinate, and colour conversion code
cleaned up some old wx->Qt calculation code
improved 'light' and 'grey' colour detection code to now work in HSV
improved colour changing code to now work in HSV
improved some internal single-shot scheduled job code
the mpv window is now plugged into the slideshow system, so when an mpv window has played its media once, a media viewer browser currently slideshow-ing will now correctly know to move to the next media
slideshows of videos should progress to the next file faster and more smoothly, with no more half-second blit of the start of the movie before the next file loads
pausing video/audio is no longer cause for the slideshow to move on--now only the 'has played once through' check will naturally trigger it. you can now more reliably seek/scrub a video during slideshow
the mpv window and native animation player is now better about pausing while a seek drag (scrub) is ongoing
various music videos that have 1 frame should now show a seekbar correctly for the mpv window, with correct timecode based seeking
fixed a bug in the video/audio resizing code that meant zoom in/out cycles would move a video player down a few pixels off center
the blank audio mpv window now has basic hardcoded zoom support and will scale down to a too-small viewer window, so you can still access the seek bar
fixed some 'has duration' calculations for audio/video that has duration but no frame count
.
shortcuts:
if a shortcut handler on an individual window does not have a double-click mapping for a command--and furthermore if none of its parent windows that are fully plugged in have one either--it will attempt to map the single-click version of the event as a backup. so now if you have an archive/delete filter, you can click fast again and the double-clicks will be interpreted as single-clicks (unless you map double-click to mean something else on the media_window or one of the media_viewer parents)
the media viewers across the program are now fully plugged into the shortcuts system for key presses, and half-plugged in for mouse clicks
'close_media_viewer' is added to all 'media_viewer' shortcut set types. enter/return/escape are now defaults for 'media_viewer' (applying to all), and middle-click/double left-click are now defaults for 'media_viewer_browser'. this is no longer hardcoded. if you are a madlad, you can now unmap all 'close_media_viewer' commands
double left-click is now assigned to 'keep' in the 'archive_delete_filter' shortcut set. due to the new click/double click rules above, this means that by default double clicking a video/audio in the archive/delete filter will now mean 'keep and move on' on that fast second click!
edit shortcut set ui now sorts its command list on load
significant shortcuts refactoring
general shortcuts code and debug code cleanup and improvement
.
the rest:
local file import pages and most downloaders will now report more file import steps to their status labels. most will blit by too fast to see, but if it hangs for a bit, you will now see the step it is caught up on. I imagine in most cases, this will be metadata generation for large videos
fixed a variety of searches that could return files not filtered to the current file domain (e.g. files in trash while in my files, or not in ipfs while in ipfs) when the search did not include an inclusive tag
updated the default danbooru file page parsers to get their new tag format, thank you to a user for submitting these
a popup message now appears while sessions are loading. it auto-dismisses once the load is complete
the edit media view options dialog (under options->media, launched from the filetype list) is now better at disabling non-applicable widgets based on filetype
fixed an issue where clicking from the autocomplete dropdown floating window to the same control's text input could result in a single flicker-frame where the dropdown is hidden
tightened the size of the splash screen white background. figuring out appropriate colour from the current stylesheet remains elusive lmao
cleaned up and wx->Qt converted a variety of event handling code
updated some 'mime' labels across the program to the new 'filetype' wording
the shortcut system now supports mouse double-clicks--left, right, or middle
the shortcuts system now differentiates between press or release single mouse clicks--although complete support for release mouse events may be a bit patchy, as full mouse integration is ongoing
the shortcut edit ui is now simpler--the command type is selected by a list, and the individual command sub-panels hide and show as appropriate--no more stupid 'set command' buttons
the shortcut edit ui now has a 'restore defaults' button that will restore an individual set back to default settings
two new shortcut sets are added--'media_viewer_media_window' and 'preview_media_window'. they control pause, pause/play, open_externally, and close/launch_media_viewer respectively. they work on the static image viewer, the native animation widget, and the new mpv player, and they support mouse clicks. the old pause/play (formerly left-click) and open_externally (double left-click) commands are no longer hardcoded
by default, the preview window's media window now launches the media viewer on a middle- or double-left-click
'media_viewer_browser' shortcut set now has 'release right-click' bound to 'show_menu', a new command, which is no longer hardcoded
most menus across the program can now be opened with the keyboard context menu key
the 'global' shortcut set now has 'exit_application', 'exit_application_force_maintenance', and 'restart_application' commands
fixed the rating increment/decrement command option not hiding in non-'media' shortcut sets
fixed some issues loading edit ui for shortcuts with rating actions
significant refactoring and some cleaning of shortcut code
.
the rest:
mpv windows should not longer get a single frame of previous-window-stretch when flicking between one mpv media to another with a different aspect ratio on the same media canvas. when a video is caught in a frame of loading, it should now flicker a frame of black
switching from one static image or native animation to another of the same type _should_ be less likely to do a single frame of stretch when transitioning. when an image or animation transition is caught on a new frame, it _should_ now flicker a frame the same colour as the media canvas background
the string transformation edit panel's individual transformation rule edit panel has had some more work: much like with shortcuts, the controls now hide and show based on transformation type, the controls' text labels now change based on transformation type, and the example text now updates on any widget change. the manual 'update example' button is removed
fixed a typo that caused an error when establishing the correct mouse cursor to use over the volume control when hydrus was using PyQt5 (rather than PySide2)
in order to reduce accidental micro-drags that cause mpv load-pause issues, starting a thumbnail drag now takes more pixels and requires a smoother drag to start, let's see how it goes
improved the show/hide logic of the floating autocomplete dropdown window. it should now also reliably detect when window focus goes from the dropdown itself to another window
fixed a bug where clearing the deletion record of a deleted tag would not remove the record from the fast cache that populates thumbnail tags (making it seem on most file loads that the tag still existed). if you were hit by this previously, please hit _database->regen->a/c cache_ one time to resync the cache
relatedly, thumbnails should now correctly live-update their deleted tags on deletion record clearance updates
if mpv is not available, opening the about window will now make a popup with the actual import error trace
updated the prototype volume/mute controls on the top media viewer hover window to be a proper 'speaker' icon button for mute with a volume slider that pops up or down on mouse-over
the new volume control is on the hover window and any media that has audio
the right-click menu of the preview viewer and media viewer now have volume submenus to set mute/volume
the client now has multiple volumes and mutes:
for mute, there is a global mute which overrides everything, and the preview and media viewers have their own mutes that just apply there.
under options->audio, you can choose whether preview windows have their own separate volume value, default is yes they do
there is a new shortcut set called 'global', which applies on the main gui and the media viewer both, and which currently has actions to alter global mute. by default, ctrl+g flips global mute
after reports of unusual rendering bugs for some users, the default mpv.conf is now more barebones. more work will happen here
.
linux:
the linux release is now built on Ubuntu 18.04 (was 16.04). unfortunately, my build packager bundled in a variety of surplus libraries, so the archive has bloated somewhat--I have removed some that I am confident are not needed, but I may have made a mistake, and there are likely more that can be taken away
the linux release now comes with mpv support
please let me know if you have any errors running this build or loading mpv. early tests seem good though!
.
the rest:
the launch/exit splash screen now uses a cleaner Qt-compatible layout system. It resizes and obeys stylesheets better, colouring text and background according to current style
removed the 'has duration' text label option from 'audio and duration' options panel as it is no longer used, and renamed the panel back to just 'audio'
the string transformation edit panel's individual transformation edit panel now shows that transformation step's example string and the transformed string, which is updated by button. this edit panel will get some more love soon, including dynamic hide/show of applicable controls and live updates of the example transformation as you type
rolled Qt back from 5.14.0 to 5.13.0 on the releases, which seems to have fixed our 'event queue sometimes halts until mouse move' issue that occurs after initial mpv load. some other ui and media viewer resize jank seems to be cleared up. I dunno what happened with 5.14, and I don't suspect it as the problem nearly as much as my currently borked Qt event processing code, but rolling back seems the easiest solution for now
fixed an issue that was crashing non-windows that were able to load mpv
mpv now loads an mpv.conf from install_dir/static/mpv-conf. please feel free to swap in another conf or edit that one as you like. I would be interested in feedback
default mpv conf is now set to specifically enable some hardware acceleration to improve playback for some users, and to never load sidecar files like subtitles as this was introducing incredibly large load lag for users with large/high latency file storage
fixed a new issue where preview windows were not unloading media (particularly significant for mpv with audio) on page change and client shutdown
fixed an issue with global volume propagation to multiple open mpv widgets
.
the rest:
added two dark qss stylesheets from the user-creation github repo to the default install
when zooming out from a zoom that makes the media bigger than the media viewer canvas to a zoom where it fits, the media will now recenter. see if you like this, maybe it should be an option
to help forestall unnamespaced filename tag spam in various new-user scenarios, the 'filename' checkbox-and-namespace widget on the filename tagging options panel now initialises with 'filename' as the namespace
fixed a recent window sizing issue with the 'the client is already running' dialog not appearing
file export paths that include subdirectories that could possibly have empty text, like "[creator]/[page]", will no longer error when this is so (e.g. if a file in this case has no creator tags)--they will eliminate the subdirectory entirely, becoming "[page]". this should work for all platforms and for any nested subdirectory
fixed an issue with some fractional dataspeeds below 1KB/s displaying with many significant figures
improved some custom event handling definition code
reworked hydrus's internal object publisher/subscriber messaging system to be more Qt-happy
if the file import tagger is given a neighbouring .txt file to pull tags from that does not decode to utf-8 nicely, it now catches and reports the error more gracefully
reworded a bit of the installing help and first-start popup to emphasise that hydrus does not auto-update
added links to https://github.com/Zweibach/text/blob/master/Hydrus/PTR.md , a new guide for the PTR, to the help
removed the old 'hardcoded shortcuts' help entry, since it is increasingly irrelevent
mpv is now available and the default for all windows users
I believed I have eliminated the final reported mpv crash
mpv load and unload delays are greatly reduced. initial load still takes about half a second, but subsequent loads are now as quick as native renderers
mpv seems to work well for gif and apng
added a very simple global volume slider and audio mute checkbox to the media viewer top hover window. this was a quick patch--much better controls and shortcuts will come in future
mpv windows now properly re-show the cursor on mouse movement
unified mpv mouse press/release handling with native animation--click down now does pause/play and starts a drag event
unfortunately, in some cases embedding mpv requires overriding local OS number rendering (e.g. 1,234 vs 1.234). hydrus number rendering is now coerced to the english style with commas until we can figure out a better solution--sorry!
cleared up an issue where simple clicks on page tabs would trigger micro-page drags that were immediately cancelled. this situation was exacerbated when the page being left had an active mpv window. the flicker of page drag cursor is now gone, and some weird situations where static clicks during busy time could move a tab should be fixed
eliminated the recent issue in the media viewer where transitioning from one media type to another through navigation, particularly mpv->other, would flicker a single frame of the last 'other' media shown(!)
fixed a bug where repeated mpv views in the preview viewer could disable client file drag and drop
the bug where thumbnails may not waterfall in unless the mouse is moving after some mpv videos are loaded for a page is relieved but not completely fixed
if the preview window is collapsed and hidden, media will no longer ever load into it
fixed an edge-case bug where the mpv window would not like being told to show nothing when it was already showing nothing
wrapped mpv load errors in a basic graceful catch
fixed an issue some users had with loading mpv's dll
.
file types:
a new file metatype, 'animation', is added, for gif and apng. these are no longer considered 'image' for a variety of purposes
the filetype selection panel, which is used in system:filetype and import folder UI, has had an overhaul--it now has tristate 'mime group' checkboxes to represent a half-filled group and expand/collapse buttons to hide the tall filetype lists. individual filetype lists will start hidden unless their default value is a partially filled group
the media view options have a similar overhaul: they are now collapsed to general filetypes by default. you set view and zoom options for the generalised 'video' type under options->media, and if you want to set specific options for webm or anything else, you can add/delete those types to override the general default
the new default options for a fresh client are just for these general types. if mpv is available, video, animations, and audio now start with mpv as the default viewer. video and animation zoom is now flexible (not fixed to 50%, 100%, 200%) and will fill the media canvas
all media view options will be reset to this simple default on update! if you have specific zoom or display preferences, please reset them after the update--but you might like to play with mpv a bit first, as it renders at large and smooth zooms very well
.
the rest:
the new thumbnail right-click file selection routine will now only focus and scroll to the first member of the selection if no other members of the new selection are already in view
fixed some caching code and sped up the new select/remove menu count generation (which can lag for very large pages) by two to six times
sped up file filter counting code by about ten percent
fixed weird layout on: migrate database panel, duplicates page (left and right), edit shortcuts, edit import folder, and the filename tagging panel
fixed an issue where the media viewer's hover windows might flicker into view for one frame when the mouse moved over the center of the media viewer for the first time
fixed a media viewer shutdown issue that would sometimes lead to the first file in the list being opened in the shutting-down viewer for an instant or highlighted as the new thumb focus
the file maintenance system that queues up missing/broken files' urls for redownload will no longer re-select the download page on every new url
fixed an issue where a downloader's tag blacklist was not being applied on the child files of certain kinds of multiple-file post (such as with pixiv)
deleting a very long tag should no longer create a very wide confirmation dialog in the manage tags dialog
fixed some 'the panel grew a bit, but the parent window didn't grow quite enough and now it has scrollbars for two pixels of extra content' sizing issues
fixed some dialog sizing calculations when the parent window was borderless fullscreen
maybe fixed a rare event processing bug
improved quality of some misc data comparison code across the program
did some significant backend event/pubsub code cleanup, mostly related to getting mpv working a bit cleaner
improved thumbnail rendering time
improved smoothness of thumbnail fade animations (at least for when they are working right, ha ha!)
basic mpv support is added. it comes with the windows build this week, and is a prototype meant for initial testing. the library is optional. users who run from source will want 'python-mpv' added via pip and libmpv available on their PATH, more details in running_from_source help
took an qt-mpv example kindly provided by a user, updated it to work with the hydrus environment, and integrated it into the client as a new choosable view type under audio/video filetypes under options->media for advanced users
reworked how the 'start paused' and 'start with embed button' media viewer options work under options->media. these are now separate checkboxes, not combined with the underlying 'show action'. existing embed/paused show actions should be converted automatically to the correct new values
unfortunately, due to some python/qt/libmpv wrapper mouse interaction issues, mpv's 'on screen controller' overlay is not available
for now, left click pause/plays the mpv window, just like the native mpv window.
preview/next frame shortcuts should work for the mpv window when playing video
no volume/mute controls yet, these will come in the coming weeks, including global mute settings
updated media show and sizing code to account for mpv widgets
reworked my animation scanbar to talk to mpv, and for my mpv window to talk back to it
improved the animation scanbar to be more flexible when frame position and num_frames are not available, both in displaying info and calculating scanbar seek clicks
mpv api version added to help->about
.
new downloader objects:
thanks to a user, updated the 'pixiv artist page' url class to a new object that covers more situations. the defunct 'pixiv artist gallery page' url class is removed
added 8kun and vch.moe download support. I got started on julay, smug, and endchan, but they were a little more tricky and I couldn't finish them in time--fingers crossed, next week
.
menu quality of life:
a right-click on thumbnail whitespace will now not send a 'deselect all' event! feel free to right-click in empty space to do an easy remove->selected
remorked the tag menu layout to move less frequently used actions down:
- moved the discard/require/permit/exclude search predicate actions down
- moved 'open in a new page' below select and copy
- moved copy above select
and some misc menu layout improvement on this menu
fixed some labelling with the discard/require/permit/exclude verbs on negated tags
right-clicking on system search predicates now shows the 'copy' menu correctly
system predicates that offer easy inverse versions (like inbox/archive) should now offer the 'exclude' verb
when right-clicking on a single tag that has siblings, its siblings and those siblings' subtags will now be listed in the copy menu!
copying 'all' tags from a list menu, with or without counts, will now always copy them in the list order
across the program, all menu 'labels' (menu text items that do not have a submenu and have no associated action, like 'imported 3 years 7 months ago') will now copy their text to the clipboard. let's see how it goes
.
other ui quality of life:
across the program's UI, filetypes are now referred to with simpler terms rather than technical mimetypes. instead of 'image/jpg', it is now typically just 'jpeg'
the 'remove selected' buttons on the gallery and watcher pages are now smaller trash icon buttons
the new page chooser will now auto-dismiss if it loses focus--so if you accidentally launch it with a middle-/double-click somewhere, just click again and it'll go away
hitting enter or return on the new page chooser now picks the 'first' button, scanning from the top-left. hitting enter twice now typically opens a new 'my files' search page
added pause_media and pause_play_media shortcuts to the media_viewer shortcut set. new clients will start with space keypress performing pause_play_media
added pause_play_slideshow shortcut to the media_viewer_browser shortcut set. this shortcut is no longer hardcoded by space keypress
the six default shortcut sets now have a small description text on their edit panels
the options->media edit panels now enable/disable widgets better based on current media/preview action
added a checkbox to _options->gui pages_ to set whether middle-clicking a tag in the media viewer or a child tag manager to open a tag search page will switch to the main gui. default is false
mr bones now reports total files, total filesize, and average filesize
mr bones now loads your fate asynchronously
.
the rest:
added tentative and simple realvideo (.rm) and realaudio (.ra) support--seems to work ok, but some weirder variable bit rate formats may not, and I have collapsed the various different extensions just down to .rm or .ra
added trueaudio (.tta) audio support
fixed a bug from the recent search optimisations where a bare inbox search would not cross-reference with the file domain (so some trash could show up in a simple inbox/'my files' query)
fixed an issue with searching for known urls by url class where the class was for a third-or-higher-level domain and was not set to match subdomains (this hit 4chan file urls for a few users)
fixed the issue with 'open externally' button panel not clearing their backgrounds properly
fixed some of the new unusual stretchy layouts in the options dialog
removed overhead from subscriptions' 'separate' operation, which should stop super CPU hang when trying to split a subscription with hundreds of thousands of urls
fixed an issue where the advanced file delete dialog would not show the simple 'permanent delete' option when launched from the media viewer's right-click menu
fixed the select/remove actions for local/remote
fixed 'set_media_focus' from manage tags to correctly activate the underlying media viewer as well as set focus
stopped the 'file lookup script' status control from resizing so wide when it fetches a url
fixed a rare mouse wheel event handling bug in the media viewer
reduced db overhead of the 'loading x/y' results generation routine. this _may_ help some users who had very slow media result loading
cleaned up how the server reports a bootup-action error such as 'cannot shut down server since it is not running'--this is now a simple statement to console, not a full error with trace
improved client shutdown when a system session shutdown call arrives at the same time as a user shutdown request--the core shutdown routine should now only occur once
fixed an issue with thumbnail presentation on collections that have their contents deleted during the thumbnail generation call
misc wx->Qt layout conversion improvements
updated the github readme to reflect some new links and so on
the right-click menus from gallery and watcher page lists now provide a 'remove' option
gallery and watchers now provide buttons and menu actions for 'retry ignored'
activating a file import status list (double-clicking or hitting enter on a selection of rows) now opens the selection in a new page
file import status buttons now have show new/all files on their right-click menus
on gallery and watcher pages, the highlight, clear highlight, pause files, and pause search/check buttons are now smaller bitmap buttons
as the old default pixiv login script is completely broken, any client with this active will have it deactivated and receive an update popup explaining the situation and suggesting to use Hydrus Companion for login instead
updated the derpibooru downloader
.
search:
when search predicates are added to the active search list, they are now better able to remove existing mutually exclusive/redundant predicates:
- system:limit, hash, and similar to predicates now remove other instances of their type
- system:has audio now removes system:no audio and vice versa
- any search predicate will remove system:everything (see how you feel about this)
improved 378's db optimisation to do tag searches in large file domains faster
namespace search predicates ('character:anything' etc...) now take advantage of the same set of temporary file domain optimisations that tag predicates do, so mixing them with other search predicates will radically improve their speed
wildcard search predicates, which have been notoriously slow in some cases, now take full advantage of the new tag search optimisations and are radically faster when mixed with other search predicates
simple tag, namespace, or wildcard searches that are mixed with a very large system:inbox predicate are now much faster
a variety of searches that include simple system predicates are now faster
integer tag searches also now use the new tag search optimisation tech, and are radically faster when mixed with other search predicates
system:known url queries now use the same temporary file domain search optimisation, and a web-domain search optimisation. this particularly improves domain and url class searches
fixed an issue with the new system:limit sorting where sort types with non-comprehensive data (like media views/viewtime, where files may not yet have records) were not delivering the 'missing' file results
improved the limit/sort_by logic to only do sort when absolutely needed
fixed the system:limit panel label to talk about the new sorted clipping
refactored tag searching code
refactored namespace searching code
refactored wildcard searching code and its related subfunctions
cleaned all mappings searching code further
.
the rest:
m4a files (and m4b) are now supported and recognised as separate audio-only mp4 files. files with a single jpeg frame for their video stream (such as an album cover) should also be recognised as audio only m4a for hydrus purposes for now. better single-frame audio support, including functional thumbnails and display, is planned for the future. please send in any m4a or m4b files that detect incorrectly
the remove thumbnail menu has been moved to a new, cleaner file filtering system. it now presents remove options for different file services and local/remote when available (most of the time, this will be 'my files'/'trash' appearing when there is a mix), including with counts for all options
the select thumbnail menu is also moved to this same file filtering system. it has a neater menu, with counts for each entry. also, when there is no current focus, or it is to be deselected, the first file to be selected is now focused and scrolled to
for thumbnail icon display and internal calculations, collections now _merge_ the locations of their members, rather than intersecting. if a collection includes any trash, or any ipfs members, it will have the appropriate icon. this also fixes some selection-by-file-service logic for collections
import folders, export folders, and subscriptions now explicitly only start after the first session has been loaded (so as well as freeing up some boot CPU competition, a quick import folder will now not miss publishing a file or two to a long-loading session)
the subscription manager now only waits 15s before starting first work (previously, the buffer was 60 seconds)
rearranged migrate tags panel so action comes before destination and added another help text line to clarify how it works. the 'go' confirmation dialog now summarises tag filtering as well
tag filter buttons now have a prefix on their labels and tooltips to better explain what they are doing
the duplicate filter right-center hover window should now shorten its height appropriately when the pairs change
fixed a couple of bugs that could appear when shutting down the duplicate filter
hackily 'fixed' an issue with duplicates processing that could cause too many 'commit and continue?' dialogs to open. a better fix here will come with a pending rewrite
dejanked a little of how migrate tags frame is launched from the manage tags dialog
updated the backup help a little and added a note about backing up to the first-start popup
improved shutdown time for a variety of situations and added a couple more text notifications to shutdown splash
cleaned up some exit code
removed the old 'service info fatten' maintenance job, which is not really needed any more
misc code cleanup
updated to Qt 5.14 on Windows and Linux builds, OpenCV 4.1.2 on all builds
if a search has system:limit, the current sort is now sent down to the database. if the sort is simple, results are now sorted before system:limit is applied, meaning you will now get the largest/longest/whateverest sample of the search! supported sorts are: import time, filesize, duration, width, height, resolution ratio, media views, media viewtime, num pixels, approx bitrate, and modified time. this does not apply to searches in the 'all known files' file domain.
after identifying a sometimes-unoptimal db access routine, wrote a new more reliable one and replaced the 60-odd places it is used in both client and server. a variety of functions will now have less 'spiky' job time, including certain combinations of regular tag and system search predicates. some jobs will have slightly higher average job time, some will be much faster in all common situations
added additional database analysis to some complicated duplicate file system jobs that adds some overhead but should reduce extreme spikes in job time for very large databases
converted some legacy db code to new access methods
fixed a bug in the new menu generation code that was not showing sessions in the 'pages' menu if there were no backups for these sessions (i.e. they have only been saved once, or are old enough to have been last saved before the backup system was added)
fixed the 'click window close button should back out, not choose the red no button' bug in the yes/no confirmation dialogs for analyze, vacuum, clear orphan, and gallery log button url import
fixed some checkbox select and data retrieval logic in the checkbox tree control and completely cleared out the buggy ipfs directory download workflow. I apologise for the delay
fixed some inelegant multihash->urls resolution in the ipfs service code that would often mean a large folder would lock the client while parsing was proceeding
when the multihash->urls resolution is going on, the popup now exposes the underlying network control. cancelling the whole job mid-parse/download is now also quicker and prettier
when a 'downloader multiple urls' popup is working, it will publish its ongoing presented files to a files button as it works, rather than just once the job is finished
improved some unusual taglist height calculations that were turning up
improved how taglists set their minimum height--the 'selection tags' list should now always have at least 15 rows, even when bunched up in a tall gallery panel
if the system clock is rewound, new objects that are saved in the backup system (atm, gui sessions) will now detect that existing backups are from the future and increase their save time to ensure they count as the newest object
short version: 'remove files from view when trashed' now works on downloader thumbs that are loaded in from a session. long version: downloader thumb pages now force 'my files' file domain for now (previously it was 'all local files')
the downloader/thread watcher right-click menus for 'show all downloaders xxx files' now has a new 'all files and trash' entry. this will show absolutely everything still in your db, for quick access to accidental deletes
the 'select a downloader' list dialog _should_ size itself better, with no double scrollbars, when there are many many downloaders and/or very long-named downloaders. if this layout works, I'll replicated it in other areas
if an unrenderable key enters a shortcut, the shortcut will now display an 'unknown key: blah' statement instead of throwing an error. this affected both the manage shortcuts dialog and the media viewer(!)
SIGTERM is now caught in non-windows systems and will initiate a fast forced shutdown
unified and played with some border styles around the program
added a user-written guide to updating to the 'getting started - installing' help page
all non-menubar menus across the program now launch on click release. some previously launched on click press. a variety of related click event behaviour is cleaned up, particularly with thumbnail/tag selection on the click down. this also fixes some users' menus immediately activating the first entry on slow clicks in some ui styles
I think I fixed the annoying single-frame delayed size-down resize on media viewer hover frames when changing media!
the vast majority of old wx panel background colour hacks are removed, so custom stylesheets should now cover much more of the UI
improved the new custom style and stylesheet setting, resetting, and error handling code, particularly for not re-applying the same style or stylesheet twice, and for handling un-re-settable styles (seems to be defaults initialised by third-party OS-wide Qt style) gracefully
fixed hyperlinks not using the custom web browser launch path as set in the options
fixed the 'migrate entire db' and 'set thumb location' buttons in the migrate database dialog
fixed a typo bug when launching the url selection tree after adding an ipfs directory to download
fixed two typo bugs when editing regex favourites and simple downloader formulae
fixed an issue where custom shortcut sets could not be deleted
fixed a typo in the edit account type panel
fixed sorting the login listctrl when there are session logins mixed with non-session logins
removed some old media viewer hover window display/raise hacks
retired the 'always show hover windows' debug mode
the media viewer will no longer perform any drag calculations on anything but left-click drag
misc Qt code refactoring/cleanup
.
url searching:
the database now stores 'known url' domain information more efficiently. it will take a few moments/minutes to reshape the db when updating
system:known url's exact url search now runs extremely fast. this will only affect new predicates of this type, not those in existing sessions
system:known url's domain search now runs much faster and matches subdomains of the given domain. this will only affect new predicates of this type, not those in existing sessions
system:known url's url class search now runs much faster. this will only affect new predicates of this type, not those in existing sessions
when entering a regex system:known url predicate, the dialog will now not OK (throwing up an error dialog) if the regex is invalid
.
the rest:
the shortcut system now allows all text characters. if it has text, it should work, but it is the wild west in terms of modifier labelling. anything unusual on your keyboard like ctrl+alt+e to make æ will _display_ as ctrl+alt+æ, but the same key combination will match up in the program all correct
added shortcut actions 'pan_top_edge', 'pan_bottom_edge', 'pan_left_edge', 'pan_right_edge' to the media viewer shortcut set that will move the current image so the respective edge is aligned with the larger canvas's
added shortcut actions 'pan_horizontal_center' and 'pan_vertical_center' to do as above but center on that axis
session save now hangs the UI significantly less, whether triggered by user command or auto-saving 'last session'
saving of last/exit sessions on client close is a little faster
the call to refresh thumbnail file info (and redraw if needed) when a file is imported or has metadata-regenererating file maintenance done will now only call for files that are actually loaded, run faster per file, run faster when the client has large collections in its session, and not hang the ui thread when waiting for the new media info to arrive
like regular popups, modal popups (like those created when big vacuum/analyze jobs jump in) will now only appear if the main gui or an on-parent child has OS focus
the main gui/on-parent child OS focus test now includes misc child windows like the autocomplete results hover window
network jobs that fail for one reason or another will now be more reliably cleaned up, and their connections returned to the connection pool. this may fix the 'too many open file handles' errors some users were seeing after long term unreliable network traffic
fixed an issue where some thumbnails that were trashed or physically deleted were being removed from 'all known files' and file repository views when it was not appropriate
connection and downloader retry time options now have a wider min/max range when in advanced mode, with an accompanying warning label for the connection panel
checker options times now have a wider min/max range when in advanced mode, with an accompanying warning label
wrote a new subscription manager to better look after subscription scheduling
rather than checking every four hours or after manage subs dialog close, subscriptions now record an indication of when they are next due for work, whether that is the estimated next check time or when bandwidth is free on remaining file downloads, and launch in a fifteen-minute window around that time. delays due to previous errors or user cancels are also taken into account. this reduces background cpu and i/o greatly for clients with large subs
if a sub is paused, or all its queries are paused, it will now never be reloaded after first load until a change via the manage subs dialog
furthermore, if a single sub takes a very long time to work, the whole sublist can re-cycle if they come up due for more work before it is finished
if a sub query is DEAD but still has outstanding files to download, it will no longer automatically pause
subs now clean up more tidily if they are running on a program exit
the subscription popup now shows check/file progress based on the number of queries that appear to have pending work. instead of 'query 300/450' with 420 that aren't due, you'll get 'query 12/30'. if a query becomes due during a round of checking, another round of checking will run
if a subscription fails to load from the db, the error is handled better and no more subs will run in that boot
improved subscription startup checking logic, tightening up various paused/dead/cansync tests
improved subscription interrupt checking logic, tightening checks on global network pause and various shutdown scenarios
cleaned up some more subscription code in prep for data storage breakup
.
qt:
added experimental Qt style settings to the new options->style page! all users should now be able to set Fusion style, and perhaps some alternate OS styles. advanced users are invited to play around with QSS stylesheets (although be warned that some of hydrus's custom colour system overrides QSS, so more work is needed here), which will be extended and made user-friendly in coming weeks
fixed tab position calculations for all tab/media drag and drops for tab bars that are centered or otherwise positioned far off top-left alignment
fixed tab drag and drop event object handling for macOS. tab and media DnD is now enabled for macOS
the popup toaster can now unhide if an on-top-of-parent non-modal frame (like review services) is focused (so hitting 'process now' should show you the work)
fixed a variety of old hacky wx close-window veto tests. the 'close client?' confirmation dialog will now reliably veto a close requent on 'no'/cancel, dialog close events that are vetoed (such as closing the manage tags dialog with pending tags) will now veto more than just the first time, and several bad media viewer archive/dupe filtering cancel and end-of-window events should now work more cleanly and correctly. users who had crashes at the end of filtering may find they are stable again
as a quick patch against some multiline notes and statuses, list controls now force single-line text in all cells
list controls now tooltip all cells
fixed the shutdown splash not updating after the daemons shut down (lmao)
'modal' message dialogs, which are created by blocking maintenance tasks such as vacuum, will no longer raise the program to the foreground on creation
should have fixed the taglist vertical positioning jank that could occur in the row after a tag with a tall emoji unicode character (and also sometimes kanji/hangul)
fixed a typo bug that was throwing an error for the upnp port widget in the local client server management panel when 'allow non local connections' was checked
improved stability of bandwidth review panel bandwidth rules refresh
improved stability of review services bandwidth rules refresh
improved some dialog cleanup code
reverted a bad environment-setting change put in last week that was causing some running-from-source users trouble
misc qt code cleanup
.
the rest:
updated the default pixiv tag search downloader to one submitted by a user. it now uses their api
updated the default twitter username lookup to a downloader submitted by a user. it fetches just the media tweet feed, making it more efficient. also added (but not linked by default) is a new tweet parser that can fetch most videos using a third-party site, advanced users may wish to play with this
added a {file_id} term for file export phrases that substitutes a unique and permanent numerical file identifier
fixed an issue where idle maintenance jobs could sometimes sneak in a few milliseconds of work during certain long shut down pauses, such as while waiting for a 'should I do shutdown work?' dialog to return. program shutdown should be snappier for many users as forced startup delays in these calls will no longer trigger
added a date 'encode' string transformation rule, which takes an integer timestamp and converts it to a pretty date string. the date rules are now renamed to the clearer 'datestring to timestamp' and vice versa
fixed page parser edit panel's 'test parse' when string transformations perform pre-parsing conversion. the handling and passing of test data for all the panels here is cleaned up throughout
system:limit predicate edit panel now has a small label describing its sampling behaviour
updated the various 8chan links in the client and help to 8kun, let me know if I missed any, and added Endchan bunker link to help menu
improved some misc status text handling across the program
refactored cache and manager code into different, simpler files
disabled the failed legacy high dpi scaling mode experiment (which was scaling up thumbnails and media in an ugly way) and returned to font-size-based natural ui scaling as set by the OS. a couple of non-font things like bitmap buttons and various layout margins are too small on >100% UI scale, and the splash screen is borked again, but it looks clear again. I'll keep working on this
fixed the custom taglist at >100% UI scale, which was spacing its tags at the wrong text height. this should survive changing ui scale while the program is open and environments with multiple monitors at different ui scale
re-fixed a critical old media-viewer-close-on-video memory leak from wx code to qt code. this was also a cause for some child ffmpeg processes not being terminated
fixed the media viewer not redrawing correctly when the media size completely exceeds the canvas window size
fixed the loading of the shortcut edit panel when the shortcut set a tag
fixed some url class edit path component ui
fixed and cleaned up some 'safe window size/position' calculations that were missing out the total frame geometry, meaning some dialogs were not moving up and left enough to show entirely on screen, and dialogs with parent-dimension gravity were not calculating initial size accurately
fixed focusing on the already-open manage tags text input when you hit 'manage tags' on a canvas with a manage tags dialog already open
fixed the html formula rule edit ui actually rendering html tag labels, lmao
updated boot-password entry to use the normal hydrus text entry dialog, and fixed a hydrus password cancel not setting a 'clean' exit for the next boot
fixed page layout splitter sash positions not resetting nicely from the menu command
fixed keyboard delete in the manage urls dialog
popup message titles are now in bold
popup message titles should now multiline correctly and fill available width
the popup messages manager should now set its min/fixed width more sensibly
subscription popups now will be wider if space is available
wrote a new class to manage better asynchronous updates for future Qt ui presentation
the file, pages, and pending menubar menus, which all require a db hit to generate, now operate on this new update class. all three should update faster when able and more politely and smoothly wait when the db is busy
reduced some accidental blocking in an old ui-update routine that kicked in when it was running hard
if the media_viewer frame type is set not to remember its 'last size', it will now instantiate with a small min size
when pasting new queries into a sub, if there are more than 5 or 50 that are already in or new, they will be rendered in a more compact way in order to stop the notification dialog growing too tall
improved stability of page update, splash screen update, and perhaps pubsub update
.
new file maintenance jobs:
added a new 'check for missing files' file maintenance job, where if the file is missing and has urls, those urls will be queued up in a new url downloader for redownload. the file record is not removed, preserving archive/inbox and import time
added a new 'check for invalid files' file maintenance job that does the same deal as above with an additional expensive byte-for-byte content check if the file is not missing
added a new 'check for invalid files' file maintenance job that only cares about invalidity--if the file is present and invalid, it is moved out but the file record is not removed
.
the rest:
network jobs that receive low-bandwidth error codes from the server now use a separate wait routine (previously, they piggybacked on the connection fail retry system). they have a separate cog-menu action to override these waits
the time delay multiple for connection errors and serverside bandwidth problems are now editable under options->connection. old default was 10 seconds base, now 15 and 60 seconds respectively
updated the danbooru login script
improved the precision of the thumbnail size estimate in database migration
the alphabetisation of a url class's GET paramaters on normalise is now optional. it is a new checkbox on the url class edit panel
when a default object fails to load from a png path, a simple error is now written to the log
macOS build is useable! tab drag and drop position calculation doesn't work yet, so intra-client file DnDs and tab rearrange DnDs are disabled for now. borderless fullscreen is also disabled, feedback on this vs maximise would be appreciated
fixed a critical bug in the macOS release that was resulting in 100% CPU repaint loop for the canvas viewer when media was loaded (wew). this may have affected certain other platforms in some situations
the linux build has a variety of common library files removed, letting your OS rely on higher compatibility system defaults. this _should_ clean up font and other issues for users running on very new/old system libraries. if you cannot run 374, please let me know your distro and version and any error messages
the special linux running from source document is updated, including info about Arch and PyQt5
fixed a windows build issue that meant some animated gifs were not able to load and render correctly
fixed a precise time fetching issue for users running from source with python 3.8
high dpi scaling should have improved support. please report on bad layout issues and other artifacts
fixed creating a serialised object png when using PyQt5
fixed file save dialogs with filetype filters when using PyQt5
fixed an important menubar related memory leak
_seem_ to have fixed an important media viewer memory leak
.
qt ui fixes:
fixed pages not collecting and sorting on creation if they do not have to, which restores the 'preserve flat unsorted order' behaviour of session loads and file drag and drop page tab creations
fixed the cursor not unhiding on move in the media viewer when over an animation or static image
fixed the issue where a new thumbnail panel would double-up with the old one for half a second if a menu caused the panel swap
reworked the elided (text that cuts off...) label code to more reliably work on single lines, which fits our purposes. the network job control (esppecially on subscription popups) and top hover window should now show their long statuses without changing their parent panel's layout
updated a variety of old text-wrap-width wx-hacks texts to instead auto-fill available space
the various downloaders should now be careful about handling large status texts. if a multiline error or html page slips in to a status somewhere, your download pages' lists should no longer go nuts with very tall spam-filled status cells
hydrus->discord drag and drop should be fixed if the BUGFIX is on!
fixed page tab drag and drop to do live drag selection with 'do not follow' behaviour (this is switched by holding down shift during drag), and, in this case, got it to return to the original page's neighbour/parent once the drop is complete
fixed 'center' dialogs positioning on the center of their parent windows, rather than the center of the primary screen
fixed the hover windows not passing shortcuts up to the media viewer when not consumed
fixed some misc 'can I consume a shortcut' focus/active checking code
fixed the various hide/parents/siblings tag menu items for tags with counts
fixed the main gui and other non-dialog windows remembering their pre-maximise/fullscreen sizes if set to remember size and previously closing while maximised/fullscreened
menubar menus should now show description text in the main gui statusbar on mouseover of their items
fixed a bad menu initialisation in the canvas preview panel
fixed a little page splitter bork and improved size of preview window on initial boot
fixed the edit notes dialog when launched from the media viewer
fixed a couple of text edit issues in edit url class panel
fixed page up/down scroll for taglists
fixed page down scroll for thumbnail grid, and fixed page up/down distance
fixed thumbnails not scrolling into view if they are keyboard-selected slightly off screen but within the scroll option percentage threshold
misc layout and style cleanup
misc refactoring
.
misc:
you can now set the maximum size of duplicate filter pair batches (default 250) under options->duplicates
when an ipfs service fails to pin a file and returns no hash or the empty multihash, this is now recognised, info dumped to log, a simple popup message sent, and the job continued. this is just a patch--better error handling here will come later
if the client or server are launched with a custom temp_dir that does not exist, it will now attempt to create it (previously errored out)
fixed a clean exit after certain client boot fail error handling, and repeated cleaner exit for the server
added some new memory profiling actions to the help->debug menu
parallel subscriptions should now initialise with less of an aggresive CPU spike
if the client or server crash before the application can be launched, the crash log is now called hydrus_crash.log. if the db dir is not yet established, it will now try to find and put it in your desktop and, failing that, then your user dir
hydrus now uses Qt for its client's user interface, migrating from wx. this is thanks to a huge effort by a user, who delivered converted code for hydrus dev to finish off
a number of hacks and patches remain to compensate for old systems, which hydrus dev will slowly clean up in normal work. ui bug and layout issue reports would be greatly appreciated
shortcut storage had to be converted from fixed wx enums to an independant system. there is a small chance that one of your shortcuts, particularly if it is on the numpad, may have been converted wrong (unusual Enter/Return buttons may be hit here). if one is not working, please check what hydrus thinks it is and try re-entering it
added tentative support for 'Mode_switch' keyboard modifier, for X11 users (and perhaps some users' AltGr?)
autocomplete results can now float in a popup window in dialogs like manage tags! they'll still embed by default, but there are now separate float/embed options for 'main gui' and 'other frame' a/cs
autocomplete results can now float in linux and macOS ok!
page drag and drop now navigates as you drag, so dropping into a page of pages works by you hovering over it and then dropping in the tabbar below, inserting exactly where you want the page to be
a couple of text inputs in the program--the watcher and gallery search pages' text inputs, particularly--now use nicer 'placeholder' text, which isn't real and only shows as grey text when the input is empty
for now, moved to icons for thumbnail 'has audio/duration' indicators, rather than the custom labels
to run the hydrus client from source, qtpy is now needed. either pyside2 (default) or qtpy5 is needed. QtCharts is optional. wx and matplotlib are no longer needed
.
misc:
'archive/delete filter' now appears even when no file is focused. it also appears when no files are selected--and will apply to everything
the system predicate edit panels now support static buttons for easy one-click select for common predicates. duration, has audio, limit, and num tags now have these
system:duration and system:num tags now render a special label if they are >0 or =0
system:untagged is now removed from the normal list
fixed a critical cpu inefficiency in the file maintenance manager's new always-on maintenance, which was lagging several users' browsing sessions while it was working
fixed ctrl+mousewheel tag autocomplete results navigation to skip over multirow parent results
fixed an issue where resetting to default bandwidth rules for a network context would not update the ui properly
fixed a bug when adding a parent/sibling from autocomplete results list
the serialised png export folder now catches when a manually inputted export path's directory does not exist
reduced metadata update lag of pages with very large media collection groups
the inaccurate 'add tags based on filename' button is now called 'import with tags'
fixed a database UNIQUE issue when two duplicate gui session save calls happen within one second
the server's lock_off command now works with the Hydrus-Key header auth (rather than hanging indefinitely wew)
the server now caches hashed access keys in the session manager, in memory, to avoid a db hit on access-key based reauthentication, and in instances where this authentication requires a db hit, now cleanly provides an appropriate 'serverbusy' error
improved some media object memory management and speedy cleanup
improved boot fail graceful exit
removed a bunch of defunct flash (swf) hacks from media viewer code
bunch of misc non-qt cleanup as I went through the code
fixed a bug with rendering network credentials for human display
cleared out the ancient tag archive sync advanced help and added a stub for the new tag migration window
the selection taglist now displays the raw 'storage' tag view, before siblings are applied
added a noneable spinctrl to control how many files are shown on a petition row double-click. it samples randomly and defaults to 256
I think I fixed the issue where the petitions taglist sometimes hangs on to some old tags after a petition process event
.
the rest:
you can now customise the animation scanbar height and nub width under options->media
all users now see the number of open pages in the pages menu
added approx total session 'weight' to the pages menu. this is an early test and will do more and update more frequently in future
added add/remove tag to favourites to the taglist right-click menu
collapsed the taglist right-click menu a little, as it was getting a bit tall
added https://gitgud.io/koto/hydrus-archive-delete, a web browser archive/delete filter, to client api help
added clipboard import/export buttons to the edit tag filter panel for the new favourites
added 'open in a new page' to media viewer right-click menus, just to put the current single media in a new page
fixed the url class edit panel not initialising with the new referral options correctly
the call that publishes new subscription/import folder media to pages now does so more politely to the gui when the db is busy
subscriptions will no longer start if global network traffice is paused
the 'hard-replace siblings/parents' action under manage tags is now a local-only operation for tag repositories. clients with unusual sibling and parent application will no longer affect the repos they sync with
on program shutdown, if a daemon takes more than thirty seconds to shutdown (which can happen in odd situations, like if a subscription run is paused by global network traffic pause, leading to shutdown deadlock), the client will stop waiting and continue with other shutdown tasks
fixed an error with client 'already running' fast exit
fixed a different error with server 'already running' exit choice
updated some ffmpeg calls to fix certain OS problems
fixed a help link to todolist recommendation and added link to new ptr guidelines
the edit tag filter panel now has load/save/delete buttons at the top to manage tag filter favourites. it starts with a handful of examples
sorting thumbnails by num tags or namespaces now uses the 'single' tag display context
the 'sort by media views/viewtime' sorts now do not put the other (viewtime/views) as an implicit secondary sort, so as to better let the user's secondary sort be used
highlighting a downloader should now not be able to create a page with duplicate thumbnails
all thumbnail pages now do an addition de-dupe check when they are created with media
when a gallery page parser now adds new urls to a file import list, urls that are invalid will now be skipped (previously, they threw an error and failed the parse
fixed a bug where if a default collect is set, pages without a collect (e.g. download pages) would nonetheless initialise with collected+sorted initial media on session load
file imports now publish the same 'refresh existing media metadata' call as the file maintenance system, meaning if the import already exists in the gui session as an 'unknown thumb', it should now refresh itself correctly
if the media canvas is called to display an invalid media (due to mime mixup or a faulty parse that slips through), it should now better recognise that and skip/dump out
fixed import of videos that have 'Duration:' in their title metadata
improved the error reporting when the old options object fails to save
removed some old ratings dialog position options storage that was causing errors on certain ratings dialog ok events
url classes now support options regarding the 'referer' http header they send (their referral url). you can set an optional converter to generate a referral url based on the url class's url and choose to always use the given referrer if available, never use a referrer, use the converter if no referrer is available, or always use the converter
the network report mode now reports on referral urls used in requests
the 'quoted' referral url (a unicode workaround) is now only applied if the referral url cannot be encoded to latin-1
the janitorial petitions processing page now lets you copy tags and left/right tags of pairs with a right-click on selected checkbox rows
cleaned a little server code
improved how the server sets and releases its 'currently busy' mode
the server no longer does <5min vacuums in a backup command
added a specific 'vacuum' server POST command that forces a full vacuum
added 'lock_on' and 'lock_off' server POST commands to lock the server and shut down the db, and restart
the new vacuum, lock_on, lock_off, and a 'is server busy?' check commands are added to the services->admin menu
added 'pause and disconnect' ability to the database mainloop
added some unit tests for url classes and the new referral url conversions and server commands
cleaned some of the thumbnail banner/icon drawing code
the old tag censorship system is now replaced by a broader tag display manager that will deal with tag storage and presentation settings. this is the first step, and some of it is not yet completely functional or as efficient as intended
management for the new tag display manager can be found under services->manage tag display. you can set per-service and all-services filters for 'multiple' display like the 'tag selection' boxes and for 'single' file views, like thumbnails and the media viewer
existing censorship rules will be added to the new manager and will apply to 'selection list' and 'single media' display rules
censorship/display rules no longer apply to underlying storage views, which are unfilterable for now, so the manage tags dialog and the autocomplete lookup will now show all tags
page and media viewer taglists now have new right-click menu options for hiding--they will provide hide options for the specific tag clicked, and its namespace more broadly, and will apply immediately to that kind of taglist (previously, this was just the tag, and launched the tag censorship edit panel)
all 'tag manager' objects behind every media object in the client now pre-compute cache layers for different tag presentation contexts. operations such as sibling collapse are now only done on file load or new siblings
for now, initial media load will take slightly longer, but various tag display updates and autocomplete tag fetches on media will be faster
changes to siblings and the new tag display rules will now trigger a reliable (although, for siblings, delayed by a few seconds) and complete tag list and thumbnail and media viewer refresh
changes to tag presentation will now correctly update collection thumbnails
some complicated sibling display and counts are now more precise
cleaned up some tag/siblings/thumb refresh notification code
cleaned up all tag manager access code
cleaned up a variety of related tag fetching, counting, and display code
.
the rest:
added 'system:modified date'. it works just like system:time imported
files with duration but no audio will now have a ' ▶ ' label in the top-left of their thumbnails, like the 'has audio' one. you can edit this label under options->audio and duration. I don't really like how this looks, so maybe we'll go to icons. let me know what you think
fixed an iteration timing bug in the new asynchronous repository mappings processing that meant large lists of mappings within an update object may be occasionally truncated, leaving some mappings unprocessed. this would more affect users on slower machines running 'process now'
any tag repository content updates issued in the last eight weeks will be scheduled for reprocessing to cover the above issue and fill in gaps for most user situations. since the vast majority of the data was added as intended, they should catch up very fast
added two pixiv url classes for their new url format
the edit subs panel now recommends users break up subs with >200k urls
the 'separate' subscriptions button now has a new 'break in half' option. subscriptions that have more than 100 queries will auto-choose this to separate
the 'quality info' button for advanced users' edit subscription panels now gives the option to additionally copy the info to your clipboard in CSV format
the lists on the gallery downloader, thread watcher, subscription, and subscriptions panels now sort their progress column by ( y, x ) (given a total status of x/y). previously, this was preceded by a percent-done sort
the hydrus network engine now recognises 429 bandwidth responses
on 429 or 509 bandwidth responses, network jobs will now go through the regular reconnection delay loop and try again later (previously, they just failed)
added 'tag migration' to 'services' menu for quick-launch
tag migration's 'go' action now skips the second confirmation if you are in advanced mode
expanded the 'reset' review services button (only visible to advanced users) to allow 'softer' resets that simply reprocess definition/content without deletion (a 'filling in the gaps' command)
fixed the 'process now' review services button disable check, which was being overzealous
cleaned up some of the new repo 'caught up' checking code
improved stability of review services 'refresh account' call
the client api /manage_pages/get_page_info call now returns a list of hash_ids beside the list of hashes, in simple or not simple mode
fixed a bug where tag import options that still had a secret deleted service reference were causing tag-parse errors on import jobs
fixed some other places that were not handling service disappearance neatly
added a note to the install/backup help to mention not to use continuous cloud-sync backups on your live db directory
the file maintenance manager now works continuously in the background, optionally in both idle and active time, with two different throttles, which are now always active
as usual, the default throttles are low-load (1 heavy job every 2 (idle) or 20 (normal) seconds), so as not to interfere with your browsing or other programs--feel free to speed them up as you wish
the options for file maintenance under 'maintenance and processing' are updated, and quick-pause actions are now available under database->maintain->file maintenance
the file maintenance manager no longer works on shutdown
the file maintenance manager will now only make a popup if it is started by the user--it otherwise now works silently in the background
the file maintenance manager now weights its jobs, so quick jobs will run faster and heavy jobs will space out more. exact weights, if you are interested, are now under the 'see description' button on the maintenance panel
file maintenance jobs now report to the debug file report mode
improved some misc file maintenance code, particularly with how the panel talks to the manager
media with new metadata will now refresh their thumbnails (for now, this means updating the has_audio icon)
.
modified timestamps:
the client now records file modified timestamps of all file imports!
on update, the retroactive population of this data for all existing local files will be scheduled on the file maintenance system, which has a new job type for this
the modified time now appears on a file's information lines that present on a right-click
the modified time can be sorted with the new 'file: modified time' sort
.
the rest:
added lexicographic sort by subtag (ignoring namespace) to the normal taglist sort selection
reworded the sort by lexicogrphic (grouped by namespace), to be (group unnamespaced)
the export files panel now has an explicit button to change the neighbouring .txt file tag services
on duplicate merge action options panel, 'sync archive' is no longer disabled for advanced users' 'alternates' duplicate action
split the download and process sync components of repositories a little
added a 'download now' button to repositories' review services panels, to hurry up metadata/update download when possible
the 'process now' button's enable/disable states should now be more reliable
the 'refresh account' button now disables when a repository is paused
improved stability of 'process now' button post-job updating
added a subscription option to the downloading option panel to change how many file-fails in a run will cause a sync to stop working early
re-added the truncated image loading mode to the debug->data actions menu. this has hung indefinitely with some bad files, so it not on by default
fixed an issue with copying an external local booru url with a upnp port
fixed an unrecoverable ui hang when a modal popup wants to self-terminate while a child yesno is open
if on a hydrus request the session key is invalid (due, for instance, to a recent serverside session clearing :^)), the session key cookie will now correctly be cleared clientside so a new one can be generated automatically on the next request
hydrus services can now take the access key as their credential using the 'Hydrus-Key' header. more options will come here, basically the same as the client api
network jobs waiting on a login process now continue faster once the login is complete (5s sleep cycle down to 1s)
perhaps fixed some linux problems with tag migration panel, perhaps not
caught and silenced a rare unimportant services shutdown error
you can now add additional local tag services under services->manage services!
new local tag services will appear in manage tags and tag import options and so on, just like when you add a tag repository
you can also delete local tag services, but you must have at least one
the default local tag service created for a new client is now renamed from 'local tags' to 'my tags'. any existing user with their local tag service called 'local tags' will be renamed on update to 'my tags'
.
ptr migration:
the ptr has been successfully migrated to user management! hydrus dev is no longer involved in running or administering it. the old bandwidth limits are removed! it has the same port and access key, but instead of hydrus.no-ip.org, it is now at ptr.hydrus.network
on update, if you sync with the ptr, you will get a yes/no asking if you want to continue using it at the new location. on yes, it'll update your server's address automatically. on no, it'll leave it as-is and pause it. if you still have a connection to my old read-only file repo, that will be paused
changed the auto repo setup command to be _help->add the public tag repository_. it points to the new location
as repo processing and related maintenance is now nicer, and secondarily since bandwidth limits are less a problem for the ptr specifically, the default clientside hydrus bandwidth limit of 64MB/day is lifted to 512MB/day. any users who are still on the old default will be updated
updated the help regarding the public tag repository, both in general description and the specific setup details
a copy of the same sanitized and frozen PTR db used to start the new PTR, and convenient tag archives of its content, are now available at https://mega.nz/#F!w7REiS7a!bTKhQvZP48Fpo-zj5MAlhQ
.
the rest:
fixed a small bug related to the new 'caught up' repository mechanic for clients that only just added (or desynced) a repository
rewrote the tag migration startup job to handle specific 'x files' jobs better--they should now start relatively instantly, no matter the size of the tag service
on 'all known files' tag migrations, a startup optimisation will now be applied if the tag service is huge
fixed the tag filter's advanced panel's 'add' buttons, which were not hooked up correctly
the internal backup job now leaves a non-auto-removing 'backup complete!' message when finished
on update, server hydrus repositories will collapse all their existing content timestamps to a single value per update. also, all future content uploads will collapse similarly, meaning all update content has the same timestamp. this adds a further layer of anonymity and is a mid-step towards future serverside db compaction (I think I can ultimately reduce server.mappings.db filesize by ~33%). if you have a tag repo with 10M+ mappings, this will take some time
hydrus servers now generate new cert/key files on boot if they are missing. whenever they generate a new cert/key, they now print a notification to the log
misc help fixes and updates, and removed some ancient help that referred to old systems
corrected journalling->journaling typo for the new experimental launch parameter
added htpa and tag service sources for parents/siblings migration that support filtering for the left and right tag of each pair
added htpa and tag service destinations for parents/siblings migration
added unit tests for all parent/siblings migration scenarios
misc improvements to mappings migration code
reworded some of the tooltip/tag filter message text to more clearly explain how the filter applies to migrations
the tag filter edit panel now has a 'test' area where you can put in an example tag to see if it passes or is blocked by the current filter
.
the rest:
fixed an issue with auto-no-ing yes/no dialogs throwing errors on exit. I am sorry for the inconvenience!
thumbnails now show the 'has audio' string on their thumbnails
'sort by file: has audio' added!
icons drawn on thumbnails are now adjusted to sit inside the border
added increment/decrement numerical ratings actions for media shortcuts! if a file hit by this action has no rating, it will initialise with 0/1 stars or max stars. please forgive the ugly expanding ui in the shortcuts panel here--I'll rewrite this to layout more dynamically in future
client repository services now track whether they are 'caught up' to their repos, which for now means processed up until at least two weeks ago, and will prohibit uploading new content until the client is caught up
repository review services panels will now display the 'caught up' status below the 'processed' progress gauge
repository review services panels will no longer duplicate 'account' status problems in the 'this client's network use' status line--both lines now refer to service/account functionality separately
repositories will now put in 'unknown error' when an empty error reason slips through the 'no requests until x time' reporting process
the new thumbnail and media viewer right-click menus now collapse the selection info lines at the top to just the top line and places all the rest (and in complicated file domain situations, this can be a long list) in a submenu off that line
the new thumbnail 'remove' submenu has separators after 'selected' and 'all' to reduce misclicks
reworded a couple of things in the manage shortcuts panel to be more clear
added wildcard support ('eva*lion') and namespace wildcards (like 'character:*') to the advanced OR text input parsing
fixed a rare issue with the duplicate filter being unable to go back or retreat from an interstitial confirm/forget/cancel dialog when every pair in the current batch cannot be displayed (such as if at least one of the pair has been physically deleted). the filter now catches this situation, informs the user, and closes itself gracefully
added two extremely advanced and dangerous launch parameters for database access testing
wrote a unified mass-migrate pipeline to make moving large amounts of data in and out of the client more powerful and more pleasant like other recent non-interrupting changes
the advanced content update dialog is now renamed to 'migrate tags', both on the review service panel and the manage tags dialog, and has been completely revamped to reflect the new migration pipeline, which works on a location-agnostic (content type, source, filtering, destination, action) model
the advanced content update dialog is now a non-modal frame--you can keep using the client while it is open, and it will not prohibit the popup from appearing while it works
added hta and tag service mapping sources to the new migration pipeline, with hash conversion, content status filtering, file domain filtering, specific hash filtering, and tag filtering
added mappings clear deletion record action to the normal content update pipeline
added hta and tag service mapping destinations to the new migration pipeline, with appropriate choosable content action (add for HTAs, add, delete, clear deletion record for local tags, and pend, petition for tag repositories)
added stub list sources and destinations for testing the new migration pipeline
wrote comprehensive unit tests to test mappings migrations with hta source and destination, including tests for file domain, hash type conversion, and tag filtering
wrote comprehensive unit tests to test mappings migrations with local tags or tag repository service source and destination, including tests for file domain, hash type conversion, tag filtering, and content status
adjusted the tag archives to have an optimise call separate from the commit call, so you can do several big jobs in a row faster, or pre-optimise, or top up a well-optimised db without wasting time re-optimising every commit
added a Close method to Tag Archives just for a nicer explicit exit
wrote a HydrusTagPairArchive to store sibling and parent tag pairs
deleted old tag db migration and archive syncing code, which is no longer used
deleted some related old hta import/export code
removed the 'tags' export option from the thumbnail share->export menu
accessing 'migrate tags' is no longer gated by advanced mode on the manage tags dialog cog menu
the hydrus database now creates a name.temp.db database while it is running to support long-term temp jobs such as the new migration. this file is otherwise unimportant and is deleted on a clean exit
.
thumbnail menu rework:
shifted the thumbnail menu around to group 'view' vs 'action' commands together more sensibly and bury less frequent commands away from the top list
the 'remove' menu command is now a submenu with filters for 'selected/not selected/ all/inbox/archive'. it also displays on a right-click with no focused file
'delete' now has separator on both sides to reduce accidental clicks
'share->open' is now moved up a level and inherits 'open in new page' and 'open externally'
the 'remote services', 'file relationships' and 'regenerate' submenus are moved to the 'manage' submenu
made similar changes to the media viewer menus, including grouping the zoom/fullscreen/slideshow commands together and making the zoom commands a submenu
misc wording and count changes to this menu
.
the rest:
added 'main_gui' shortcuts for 'refresh_all_pages' and 'refresh_page_of_pages'. the second shortcut refreshes all pages under the most immediate page of pages parent
fixed an issue where under-construction OR predicates were not displaying with the system predicate list
the 'import local files' frame, which pops up when you drop some paths on the client, is now on the new panel system and sizes and cleans itself up more sensibly as a result
refactored the remaining 100-odd yesno dialogs to the one that works on the new panel system
misc yesno dialog message and logic cleanup
fixed an issue where the export files dialog could hang indefinitely if the filename phrase involved long shared tags that resulted in duplicate paths for the first 245 characters in length. now, the long-filename truncation is done before the de-duping ' (n)' text is appended, and the length limit is reduced to 240
removed the now-defunct 'regen similar files metadata' command under the database menu--this is now handled in the new file maintenance processing system
improved database optimisation code to better check if empty/small tables have suddenly grown and improved quality of optimisation data for frequently emptying tables
fixed the 'clear orphan file records' maintenance task, which was not performing the final clear on the correct domain
fixed some bad error presentation
fixed the duplicate page not refreshing maintenance numbers after maintenance job completion
cleared out a bunch of old py2to3 safety code, maybe sped up some sibling/parent/mappings stuff
improved the new asynchronous repository processing system in several ways:
- it now uses the time it is allotted more accurately. when it has 0.45s to work, it hits this mark more often, especially on slower machines
- it is now politer to the ui if plenty of other work is going on--if the db is loading search results or you are viewing video, it should pause for milliseconds as needed
- it can now work up to 90% of the time during a manual 'process now' run
- when it is working faster than expected, it accelerates its load to operate more efficiently
as a result, the new system should now have faster rows/s and lag out the ui less
.
client api:
improved how parameters are fetched and tested against expected type and given default values if appropriate, and updated all client api code to use this new system
added /manage_pages/get_page_info, which gives simple or detailed info on a given page, found by page_key
added page info response to hdd importers, simple downloaders, url downloaders, gallery downloaders, watcher downloaders--they say their pause status and file/gallery import info as appropriate
added page info response to file import caches--they say their status and num_done/num_to_do, and in detailed mode report file import item info, which is url/path, created, modified, and source times, current status, and any note
added page info response to gallery logs--they say their status and num_done/num_to_do, and in detailed mode report gallery log info, which is url, created and modified times, current status, and any note
added page info response to thumbnail panes--they say their total num files, and in detailed mode list their ordered hashes
started some help for this expansion, but it will need some feedback and more work to finish
the client api now sorts /get-files/search_files results by import time, newest to oldest. this first hardcoded sort comes to help implement booru-like pagination, but will be expanded to support more types as I flesh out the ui side (as below) as well
hydrus services, including the client, should now be able to handle larger request header+path total size (16KB->1MB). this helps some larger GET queries in the client api. let's see how this goes
client api is now version 11
.
collect improvements:
the collect data attached to pages is updated to its own object. the default value and existing page settings should update. all ui now handles the new clean object, rather than the old messy list
the new collect object supports an option for whether to collect 'unmatched' thumbs or to leave them separate. this displays in the ui as a dropdown beside the collect-by checkboxlist
to better distinguish between unmatched singles and matched collections with just one item, all one-item collections will now act as collections, with the little '1' collection icon in their corner (previously, they were split into singles). if this is annoying, I will add another option to control whether this occurs
removed some old 'integrate media into existing collected structure code' that was complicated, never used, and now broken
misc sort/collect refactoring
deleted some old unused collection code
.
the rest:
entering tags in the filename tagging panel, either for all or just selected, now pushes those tags to the 'recent tags' list in later manage tags dialogs
added a framework to start sorting search results before the system:limit is applied--I will soon extend this to start catching the current ui sort (say, 'largest files first', and cut a system:limit appropriately, rather than the current random sample)
added a faster table size check on the analyze maintenance call that will recognise fast-growing tables (e.g. initially empty/tiny repository processing tables that may have seen a ton of recent work) and schedule them better (this was previously potentially hanging certain maintenance checks/shutdown by several minutes when hitting a surprisingly giant table)
reduced the analyze frequency for established tables
the client will now explicitly count as 'idle' and 'very idle' during shutdown maintenance time, in case any shutdown job is considering that for how greedy it should be with work time
fixed an issue where appending new media (thumbnails) to a page that already had that media but within a collection could create a duplicate media entry and invalidate some internal data links to the old media
subscriptions will no longer print full traceback information when a network error causes a sync fail
updated to yet another deviant art file page parser. title tags and embedded image links should be fixed again, post/source time is not fixed
the deviant current art login script is confirmed to work for some users. my guess is certain people are getting cloudflare blocked or aren't being shown the new login page all the time yet, please send in any more info you discover
the client will now recover from a missing options object by putting in a fresh one with default values, including a popup notifying you of the error and giving you a chance to bail out
added a warning and link to the quicksync to the access_keys help page
if the os commands the client to close due to a log off or system shut down, the client will kindly ask for a bit more time do to so if it is available
repository processing is now no longer a monolithic atomic database job! it now loads update files at a 'higher' level and streams packets of work to the database without occupying it continuously! hence, repository processing no longer creates a 'modal' popup that blocks the client--you can keep browsing while it works, and it won't hang up the client!
this new system runs on some different timings. in this first version, it will have lower rows/s in some situations and higher in others. please send me feedback if your processing is running significantly slower than before and I will tweak how this new routine decides to work and take breaks
multiple repos can now sync at once, ha ha
shutdown repository processing now states the name of the service being processed and x/y update process in the exit splash screen
the process that runs after repository processing that re-syncs all the open thumbnails' tags now works regardless of the number of thumbnails open and works asynchronously, streaming new tag managers in a way that will not block the main thread
'process now' button on review services is now available to all users and has a reworded warning text
the 1 hour limit on a repo processing job is now gone
pre-processing disk cache population is tentatively gone--let's see how it goes
the 10s db transaction time is raised to 30s. this speed some things up, including the new repo processing, but if a crash occurs, hydrus may now lose up to 30s of changes before the crash
.
the rest:
users in advanced mode now have a 'OR' button on their serch autocomplete input dropdown panels. this button opens a new panel that plugs into prkc's neat raw-text -> CNF parser, which allows you to enter raw-text searches such as '( blue eyes and blonde hair ) or ( green eyes and red hair )' into hydrus
fixed the silent audio track detection code, which was handling a data type incorrectly
improved the silent audio track detection code to handle another type of silence, thank you to the users who submitted examples--please send more false positives if you find them
fixed an issue where thumbnails that underwent a file metadata regeneration were not appearing to receive content updates (such as archive, or new tags/ratings) until a subsequent reload showed they had happened silently. this is a long-time bug, but the big whack of files added to the files maintenance system last week revealed it
the 'pause ui update cycles while main gui is minimised' change from last week now works on a per-frame basis. if the main gui is minimised, media viewers that are up will still run videos and so on, and vice versa
a few more ui events (e.g. statusbar & menubar updates) no longer occur while the client is minimised
duplicate processing pages will now only initialise and refresh their maintenance and dupe count numbers while they are the current page. this should speed up session load for heavy users and those with multiple duplicate pages open
gave the new autocomplete 'should broadcast the current text' tests another pass--it should be more reliable now broadcasting 'blue eyes' in the up-to-200ms window where the stub/full results for, say, 'blue ey' are still in
fixed an accidental logical error that meant 'character:'-style autocomplete queries could do a search and give some odd results, rather than just 'character:*anything*'. a similar check is added to the 'write' autocomplete
fixed an issue with autocomplete not clearing its list properly, defaulting back to the last cached results, when it wants to fetch system preds but cannot due to a busy db
fixed GET-argument gallery searches for search texts that include '&', '=', '/', or '?' (think 'panty_&_stocking_with_garterbelt')
removed the pixiv login script from the defaults--apparently they have added a captcha, so using Hydrus Companion with the Client API is now your best bet
the client's petition processing page will now prefer to fetch the same petition type as the last completed job, rather than always going for the top type with non-zero count
the client's petition processing page now has options to sort parent or sibling petitions by the left side or right--and it preserves check status!
the client's petition processing page now sorts tags by namespace first, then subtag
the client now starts, restarts, and stops port-hosted services using the same new technique as the server, increasing reliability and waiting more correctly for previous services to stop and so on
the client now explicitly commands its services to shut down on application close. a rare issue could sometimes leave the process alive because of a client api still hanging on to an old connection and having trouble with the shut-down db
the file maintenance manager will no longer spam to log during shutdown maintenance
sketched out first skeleton of the new unified global maintenance manager
improved some post-boot-error shutdown handling that was also doing tiny late errors on server 'stop' command
added endchan bunker links to contact pages and github readme
wrote a detection routine that can determine if a video has audio. it reads actual audio data and should be able to detect videos with a 'fake' silent audio track and consider them as not having audio
extended the client database, file import pipeline, and file metadata object to track the new has_audio value
flash files and audio files (like mp3) are considered to always have audio
all 'maybe' audio files (atm this means video) are queued up for a file metadata reparse in the files maintenance manager. your existing videos will start off as not having audio, but once they are rescanned, they will get it. this is one of the first big jobs of the new maintenance system, and I expect it will need some different throttling rules to finish this job in reasonable time--by default it does 100 files a day, but if you have 50,000 videos, that's a long time!
files now show if they have audio in their info string that appears on thumbnail right-click or the top of the media viewer. it defaults to a unicode character, but can be edited under the new 'sound' options page
added a system:has audio predicate to search for files with/without audio
updated file import unit tests to check 'has audio' parsing, and added tests for system:has audio
.
client api:
the /get_files/file_metadata call now provides has_audio info
the /get_files/file_metadata call now provides known_urls!
added 'cookie management' permission
added /manage_cookies/get_cookies to get current cookies by domain
added /manage_cookies/set_cookies to set or clear current cookies
added/updated unit tests for the above
updated help for the above
client api version is now 10
.
the rest:
system:hash and system:similar to now accept multiple hashes! so, if you have 100 md5s, you can now search for them all at once
the thumbnail right-click->file relationships->find similar files now works for multiple selections!
when system:hash was just one hash, it would run before anything else and complete a search immediately on finding a match, but now it works like any other predicate, checking for file domain and ANDing with other predicates in the search
the 'complete' file maintenance regen job now only does file metadata, not a complete thumb regen. its name and labels are updated to reflect this, and any existing job in the system will get the separate thumb regen job
the file maintenance manager now has a couple of how-to sentences at the top, and a new 'see description' button will talk more about each job type
the login script testing system now uses a duplicate of the existing domain manager (rather than a fresh empty one), so it will inherit current http headers such as default User-Agent, the lacking of which was messing up some tests
fixed the login script testing system not showing downloaded data
subscriptions with multiple queries now publish the files they have imported as soon as each query has finished, rather than waiting for the whole sub to be done
subscriptions now publish the files they have imported to page/popup even if they have an error
added 9:16, 2:3, and 4:5 to the duplicate comparison statement system, for various vertical social media types
the autocomplete tag search 'read', which appears on places like search pages, should now more reliably accept the current entered text when there are no search results yet to show
the autocomplete tag search 'write', which appears on places like the manage tags dialog, should now correctly accept the input (including appropriate sibling-collapse) when you select a 'stub' result while other results are still loading, rather than broadcasting the exact current text
fixed the deviant art file page parser to get source time--however the login script may now be broken/unreliable
fixed a missing dialog import when deleting a string transformation
reduced the base network connection error reattempt time to 10s (from 60s). there will be more work here in future
network jobs that are waiting on a connection error now have a reattempt wait override option in their cog icon menus
the post-bad-shutdown 'open your default session or a blank page' dialog will now auto-choose to open your default session in 15 seconds
a variety of ui-update events will now not fire as long as the main gui is minimised. as well as saving a sliver of resources, I believe this may fix an issue where long-running subscriptions and other import pipelines could sometimes put the ui in an unrecoverable state due to too many thumb-fade etc... events when the currently focused page was receiving new files while the main gui was minimised
maybe fixed a rare problem with deleting old pages
updated the duplicates help text and screenshots to reflect the new system
duplicate files search tree rebalancing is now done automatically on the normal idle maintenance routine, and its over-technical UI is removed from the duplicates page
the duplicate filter's resolution comparison statement now specifies 480p, 720p, 1080p, and 4k resolutions and highlights resolutions with odd (i.e. non-even) numbers
if the files are of different resolution, a new 'ratio' comparison statement will now show if either have a nice ratio, with current list 1:1, 4:3, 5:4, 16:9, 21:9, 2.35:1
added a 'stop filtering' button to the duplicate hover frame
made the ill-fitting 'X' button on top hover frame a stop button and cleaned up some misc related ui layout
added a 'remove this file's potential pairs' command to the thumbnail file relationships menu
if in advanced mode, multiple thumbnail selection right-click menus' file relationships submenus will now offer mass remove/reset commands for the whole selection. available commands are: 'reset search', 'remove potentials', 'dissolve dupe groups', 'dissolve alt groups', 'remove false positives'
.
the rest:
added link to https://gitgud.io/koto/hydrus-dd/ , a neat neural net tagging library that uses the DeepDanbooru model and has several ways of talking to hydrus, to the client api help
cleaned up a little of the ipfs file download code, mostly improving error/cancel states
rewrote some ancient file repository file download code, which ipfs was also using when commanded to download via a remote thumbnail middle-click. this code and its related popup is now cleaner, cancellable, and session-based rather than saving download records to the db (which caused a couple of edge-case annoyances for certain clients). I think it will need a bit more work, but it is much saner than it was previously
if you do not have the manage tags dialog set to add parents when you add tags, the autocomplete input will no longer expand parents in its results list
fixed an issue displaying the 'select a downloader' list when two GUGs have the same name
hitting apply on the manage parsers or url classes dialogs will now automatically do a 'try to link' action as under manage url class links
fixed (I think!) how the server services start, which was broken for some users in 361. furthermore, errors during initial service creation will now cancel the boot with a nice message, and the 'running ... ctrl+c' message will appear strictly after the services have started ok the first time, and services will shut down completely before the db is asked to stop
improved how the program recognises shutdowns right after boot errors, which should speed up clean shutdowns after certain bad server starts
the server will use an existing server.crt and server.key pair if they exist on db creation, and complain nicely if only one is present
the 'ensure file out of the similar files system' file maintenance job result will now automatically remove from/dissolve the file's duplicate group, if any, and clear out outstanding potential pairs
a system language path translation error that was occuring in some unusual filesystems when checking for free disk space before big jobs is now handled better
like repository processing, there is now a 1 hour hard limit on any individual import folder run
fixed an issue where if a gallery url fetch produced faulty urls, it could sometimes invalidate the whole page with an error rather than just the bad file url items
subscriptions will now stop a gallery-page-results-urls-add action early if that one page produces 100 previously seen before urls in a row. this _should_ fix the issue users were seeing with pixiv artist subs resyncing with much older urls that had previously been compacted out of the sub's cache
until we can get better asynch ui feedback for admin-level repository commands (like fetching/setting account types), they now override bandwidth rules and only try the connection once for quicker responses
the duplicate filter now compares the pixel content of static image pairs of the same resolution--if they have the exact same pixels, a comparison statement is added, and if one file is a png and the other not (i.e. the png is likely a useless clipboard copy), the statement notes this and a strong duplicate score is applied
added 'system:is/is not best file of its group' to search for file kings
renamed 'system:num duplicate relationships' to 'system:num file relationships'
wrapped the two file relationship system predicates into one 'system:file relationships' stub predicate that opens to a dialog with two pred panels
added a 'add potential pairs' command to the thumbnail right-click file relationships menu, which will force-queue files for the duplicates filter
the duplicate filter now ensures the two medias' zoom is locked so they have the same width through a transition. furthermore, their current dragged top-left position is pinned in the same location. this ensures files that have slightly different resolution ratios (especially when they are just a couple of pixels off) still remain reasonably comparable when switching back and forth
reworked and simplified how position/drag delta is handled in the media canvas to support the above
fixed the 'custom action' button on the duplicate filter, which had no 'delete neither' choice and whose 'forget it' button cancelled the whole custom operation, making it impossible to custom action without deleting something. I have added a 'delete neither' green-text button to the front, as the default action
mr bones now reports on your potential, duplicate, and alternates numbers
.
tag autocomplete:
greatly sped up tag autocomplete search when fetching from a current media view (i.e. from thumbnails in the search page)--it had some CPU-inefficient testing/counting that mattered at high media/tag counts
greatly improved cancelability of tag autocomplete search when pulling from a current media view--this was resulting in high lag when typing fast with multi-thousand results
fixed the gui-level tag matching test to match namespaced search inputs with offset subtags (e.g. 'character:aran' now matches 'character:samus aran'), both for wildcard and specific namespaces
when typing an explicit wildcard tag search that does not end in a *, you will now be presented with two wildcard options--one with the implicit * suffix, one without
fixed 'write' tag autocomplete inputs (like in manage tags) being able to search for chunky 'namespace:*' explicit wildcard searches
.
the rest:
fixed the ipfs nocopy path translation control saving rows for client file paths outside of the main install path for non-Windows, where it was forgetting on save
renamed 'system:size' to 'system:filesize'
sped up some system:inbox searches
disabled a PIL 'load truncated images' backup mode, which on the current version can seemingly lead to infinite load hangs
file report mode now prints info when it deletes/recycles a path, including stack traces
fixed a long-running and silent 'port already running' bug related to setting services on the server that was stopping successful service-set-restart from the client in many situations. 'port is already running' checks that conflict with other processes will now give an immediate error to the client without saving any changes
the server now prints to the log as it stops/starts/has started its services
improved how the server can report certain 500 errors
the 'critical service tag/file reference' repository processing error has been improved: rather than reset the whole repository, it now pauses the repo and resets processing status for just the repo's 'definition' update files (without deleting any existing entries, so they should ultimately reprocess super fast) and also schedules a complete integrity and metadata check for all updated files
keyboard interrupts from the console should now trigger a clean exit request for the client
polite and forced shutdown requests when logging off should now trigger a fast exit (i.e. no yes/no dialog, no shutdown maintenance, but otherwise session saved and so on) for the client. this fast exit is noted in the log
moved the tag and rating service listctrls in duplicate merge options panel to the new listctrl object
moved the manage regex favourites listctrl to the new object
updated a bunch of yes/no dialogs to the new panel system
deleted some old unused dialog code and related unit tests
fixed up deletion-and-reimport file location handling for lingering media objects, which were not correctly forgetting combined local file deletion record on the reimport
improved shutdown error handling during repo processing
after various tag autocomplete async work, fetch timings get a complete overhaul this week. the intention is for a/c jobs to appear as fast as possible, with good ui feedback, without interrupting ui while they work. feedback on how this works IRL would be appreciated
there are now just two autocomplete options under options->speed and memory:
- whether autocomplete results are ever fetched automatically, defaults to true
- the max number of characters in the input that will cause just exact results vs. full autocomplete results, defaults to 2, can be None
namespaces are no longer searched from an unnamespaced query ('char' no longer matches 'character:samus aran'). this proved too slow for real use, and remains better available with explicit namespace searches such as 'character:*' or 'char*:*'
the 'exact results' character limit now also applies to subtags of namespace searches! so, entering 'character:a' will deliver the same short exact match results as just 'a'--no more gigantic lists when you put in a simple namespace
improved tag results caching to deal with the new non-namespace matching on subtag input
tag autocomplete dropdowns will now display a non-selectable 'loading results...' label when results take more than 200ms to load.
tag autocomplete dropdowns will now also display 'static' tags, such as 'namespace:*anything*' for 'read' inputs and the exact entered text and possible siblings/parents for 'write' inputs, during loading. so, typing 'character:' just to get the special 'character:*anything*' predicate is now simple and does not need a whole load wait to enter!
cleaned up some tag listbox code to handle parent selection and navigation better along with the new label type
greatly improved autocomplete search logic in the critical text search portion, collapsing it into one cleverer and more easily cancellable query rather than two or three simpler ones with potentially gigantic lists thrown back and forth
improved speed of autocomplete cancel for certain large lists with many siblings
.
file maintenance:
the new file maintenance ui now shows scheduled jobs in a listctrl, and only shows jobs that have outstanding work. you can clear/do work on multiple selected jobs
the file manager should now try to guarantee at least 644 permission on file imports (previously, it was only trying to add 600, which lead to problems with nocopy ipfs running on another user etc...)
added a file maintenance job to check and fix file permissions
added a file maintenance job to regenerate similar files metadata
added a file maintenance job to check if a file should be in the similar files system--if it should and isn't, it is queued to get its metadata data regenerated, and if it is and shouldn't be, it is removed
the previous bulky similar files metadata regen job from the duplicates page is now removed, and any outstanding scheduled regen will be transferred to the new file maintenance manager on update
.
client api:
added POST /manage_pages/focus_page, which makes the given page the current page in the main gui
added help and unit tests for this new call
client api is now version 9
.
the rest:
fixed an issue recording media viewtimes when no max viewtime is set
fixed the new missingdirectory errors not printing the missing path
fixed an issue with some human-started repository actions waiting silently on bandwidth when it was not intended (e.g. account refresh)
export folders now raise proper errors and pause themselves if their path is not set, does not exist on the file system, or is not a directory (previously, they silently stopped work without error)
cleaned up some misc import folder code, and put in additional protections to the delete/move code to ensure folders cannot be so actioned if they somehow end up in the path import queue
when unpinning a file or directory from ipfs, the clientside service now first checks that the current daemon considers it pinned (previously, this 500 errored when the object was not pinned due to a reinitialised daemon etc...)
fixed an issue with the new ipfs path translation control, which was forgetting values when the clientside path was outside of the default db structure
media objects that transition from trashed to physically deleted but remain in view will now correctly be aware of their complete previously-deleted status (rather than being simply remote, as they were before until a client restart)
improved some of the recent duplicates db update code to pre-optimise the new tables on update (some users were getting slow behaviour due to mis-scheduled analysis maintenance)
extended the new panel system to deal with custom button panels and moved the duplicate filter 'commit and continue?' dialog to the new panel system
moved the archive/delete and duplicate filter 'commit and finish' dialog to the new panel system
wrote a new question panel for the typical yes/no dialog used across the program and started a cleanup job to migrate all 140-odd instances of this over
fixed an issue where a program instance that quit due to a user deciding to leave an already running instance in place would clear the original instance's 'running' file in its shutdown, meaning subsequent runs would charge ahead and hit 'database is locked' problems on db init!
wrote a new 'similar files metadata generation report mode' to provide debug info on this cpu/gpu intensive routine
added 'why use sqlite?' entry to the help faq, with a link to prkc's excellent document about the subject, https://gitgud.io/prkc/hydrus-why-sqlite/blob/master/README.md
also added prkc's excellent Linux package requirements information to the 'running from source' help page
fixed some old py 2.7 references in running from source help and an old link in ipfs help
moved the 'file viewing statistics' menu down on the database menu
wrote a new panel to better show ipfs daemon status and added it to the review and manage ipfs service panels
added nocopy config review and enable status and buttons to this new panel
added an EXPERIMENTAL 'use nocopy' checkbox to the ipfs manage services panel
added accompanying WEWLAD path translation ui to enable nocopy when your hydrus media storage paths are inaccessible to the ipfs daemon for nocopy purposes. a help button explains this more--it currently needs some symlinking, so non-advanced users should stay away
if everything is set up, ipfs nocopy seems to work! I am not totally happy about the setup required here, so feedback from advanced ipfs-fluent users would be appreciated and we can iterate on this
improved stability of ipfs daemon/version checking code
.
file maintenance:
wrote some proper file maintenance ui under database->maintain->review scheduled file maintenance!
for existing work, the new file maintenance ui shows how much work is scheduled for each job type and lets you cancel that work or run it manually
for new work, the new file maintenance ui lets you queue up work of any type for files you select with the standard tag autocomplete search interface! you can schedule all pngs to be rescanned in case they are truly apngs, or regen thumbs for all files imported before a certain date, or whatever you wish. you can also queue up repository update files
the file maintenance manager can now deal with repository update files when it does a complete file metadata regen
the file maintenance manager now takes responsibility for checking file presence and file integrity. the old 'check file integrity' options under database->maintenance, which did all files in one go, are now gone
file integrity checks will now always export broken files and missing/broken files' known urls to .txt files to your db_dir/missing_and_invalid_files. appropriate popups and log data will be sent as well. also, the known urls will be both exported on a per-file .txt basis and appended to one unified .txt
if a file now fails to parse on a metadata reparse, it is now automatically checked for file data integrity
if a repository encounters a missing, invalid, or incorrect filetype update during update processing, it now schedules all updates in the repo to be appropriately rescanned by the file maintenance manager
if the storage subdirectory directory does not exist on a client file path request or thumbnail-add attempt, a special error will now be raised with instructions to reconnect the location or shut the client down immediately
cleaned up some ffmpeg mime-detection logspam
.
duplicates:
added several single-file thumbnail right-click dissolve/reset duplicate actions:
- reset search status
- remove from duplicate group (if in one and not the king)
- dissolve duplicate group (if in a group)
- remove from alternate group (if in one)
- dissolve alternate group (if in one)
- clear false-positive relations (if it has some)
added some new code to deal with dissolution and member extraction at the db level
when a member is extracted from alternate group, its constituent files are now requeued for potential search
multi-selection duplicate right-click actions are now available to non-advanced-mode users
wrote some unit tests for the new dissolve/reset actions
cleaned up some misc duplicates code
.
the rest:
fixed a recent bug in the file lookup script GET call--I apologise for the mistake
the main gui page tab menu now lets you sort page tabs by the number of files they have
deviant art seem to be rolling out a new page format. this week hydrus introduces completely new deviant art downloader objects that, fingers crossed, will update any existing users smoothly and also provide new tag search functionality. users who are still logged in may still be getting the old page format. if this is you, and this update does not work (although I _think_ it should, even so), please try clearing your existing login and logging in again
new deviant art login script, artist + tag GUGs, gallery url classes, file and gallery parsers
updated the shimmie file page parser to pull source time and md5
improved the 'process now' advanced button to only focus on actual specific outstanding processing. previously, it was also checking for new metadata when due, which, when the server was not available, could seemingly idle for a time before actually processing updates due to the new delaying connection retry code
wrote a new 'file import report mode' mode to help->debug->report modes
fixed a progress display issue with the janitorial petitions processing page
improved accuracy of sibling and parent petition counts, and properly capped them at 1000
mapping petitions are now grouped by namespace, and will come in more manageable chunks
fixed the server launch-and-init test debug code
misc string-to-string control improvements to support the new ipfs edit ui
removed the old 'continual tag archive sync' legacy code from tag services, which has been semi/non-functional for a long time
cleaned up the annoying separator hanging on the end of certain tag right-click menus
cleared out the 'Exception ignored in' spam that is often printed after the log closes
the final large data storage overhaul work of the duplicates work big job is done--potential duplicate information is now stored more sensibly and efficiently. potential pair information is now stored between duplicate file groups, rather than files themselves. when duplicate file groups are merged, or alternate or false positive relationships set, potentials are merged and culled appropriately
your existing potential data will be updated. the current potential pairs queue size will shrink as duplicate potential relationships are merged
the duplicate filter now presents file kings as comparison files when possible, increasing pair difference and decision value
potential pair information is now stored with the 'distance' between the two files as found by the similar-files search system. the duplicate filter will serve files with closer distance first, which increases decision value by front-loading likely duplicates instead of alts. distance values for existing potential pair info is estimated on update, so if you have done search distance 2 or greater and would like to fill in this data accurately to get closer potentials first, you might like to reset your potential duplicates under the cog icon (bear in mind this reset will schedule a decent whack of CPU for your idle maintenance time)
setting alternate relationship on a pair is now fixed more concretely, ensuring that in various search expansions or resets that the same pair will not come up again. this solves some related problems users have had trying to 'fix' larger alternate groups in place--you may see your alternates compared one last time, but that should be the final go. these fixed relationships are merged as intra-alternate group members merge due to duplicate-setting events
a variety of potential duplicates code has been streamlined based on the new duplicate group relationship
improved how a second-best king representative of a group is selected in various file relationship fetching jobs when the true king is not permitted by search domain
one critical part of the new potential duplicates system is more complicated. if you experience much slower searches or count retrievals IRL, please let me know your details
expanded duplicates unit tests to test potential counts for all tested situations
fixed a bug where alternate group merging would not cull now-invalid false-positive potential pairs
the rest:
updated the default pixiv parser to work with their new format--thank you to a user for providing this fix
fixed the issue where mouse scroll events were not being processed by the main viewer canvas when it did not have focus
file page parsers that produce multiple urls through subsidiary page parsers now correctly pass down associated urls and tags to their child file import items
updated to wx 4.0.6 on all built platforms--looks like a bunch of bug fixes, so fingers-crossed this improves some stability and jank
updated the recent server access-key-arg-parsing routine to check access from the header before parsing args, which fixes an issue with testing decompression bomb permission on file POST requests on the file repository. generally improved code here to deal more gracefully with failures
the repositories now max out at 1000 count when fetching pending petition counts (speeding up access when there are large queues)
the repositories now fetch petitions much faster when there are large queues
frames and dialogs will be slightly more aggressive about ensuring their parents now get focus back when they are closed (rather than the top level main gui, which sometimes happens due to window manager weirdness)
rewrote a bad old legacy method of refocusing the manage tags panel that kicks in when the 'open manage tags' action is processed by the media viewer canvas but the panel is already open
hitting 'refresh account' on a paused service now gives a better immediate message rather than failing after delay on a confusing 'bad login' error
improved login errors' text to specify the exact problem raised by the login manager
fixed a problem in the duplicates page when a status update is called before the initial db status fetch is complete
the manage tag siblings panel now detects if the pair you wish to add connects to a loop already in the database (which is a rare but possible case). previously it would hang indefinitely! it now cancels the add, communicates the tags in the loop, and recommends you break it manually
added a link to https://github.com/cravxx/hydrus.js , a node.js module that plugs into the client api, to the help
a variety of user-started network jobs such as refreshing account and testing a server connection under manage services now only attempt connection once (to fail faster as the user waits)
the 'test address' job under manage services is now asynchronous and will not hang the ui while it waits for a response
fixed some unstable thread-to-wx code under the 'test access key' job under manage services
improved some file handling to ensure open files are closed more promptly in certain circumstances
fixed some unstable thread-to-wx communication in the ipfs review services panel
improved the accuracy of the network engine's 'incomplete download' test and bandwidth reporting to work with exact byte counts when available, regardless of content encoding. downloads that provide too few bytes in ways that were previously not caught will be reattempted according to the normal connection reattempt rules. these network fixes may solve some broken jpegs and json some users have seen from unreliable servers
fixed watcher entries in the watcher page list not reporting their file and check download status as they work (as the gallery downloader does)
the client api will now deliver cleaner 400 errors when a given url argument is empty or otherwise fails to normalise (previously it was giving 500s)
the client api can now receive the access key through a GET or POST parameter rather than the header
the client api now supports GET /session_key, which provides a temporary key that gives the same access as its permanent access key with the Hydrus-Client-API-Session-Key name through header or GET/POST param. it expires after 24 hours of inactivity or if the client is restarted
the GET /manage_pages/get_pages call now returns the unique 'page_key' identifier that will be useful in future page management when multiple pages share a name
the POST /add_urls/add_url command now takes 'destination_page_key' to exactly specify which page you would like a URL to end up on. if the page is not found, or it is the incorrect type, the standard page selection/creation rules will apply
cleaned up some serverside request processing code
cleaned up some misc client api permission checking code
updated client unit tests to check the new changes
updated client api help to reflect the new changes
cleaned up some GET and POST parameter parsing
client api version is now 8
.
shortcut and hover window fixes:
moved the canvas shortcut processing code more towards the new shortcut system
the OS X shortcut-in-media-viewer issue, which was being boshed in a similar way to the main gui last week, should now be fixed
when the hover windows have focus, they now pass shortcuts up to the canvas parent more reliably
removed a legacy menu highlight-tracking system that was malfunctioning and generally throwing a slow-memory-leaking wrench in several places, particularly some non-Windows situations
the 'menubar is open' test code is now only active for Windows. the other platforms have mixed reliability with menubar open/close events
some related OS X hover-window flickering and hiding-under the main page problems (having problems due to thinking menus were open) are also fixed
some hover window flicker on certain focus changes due to clicking focus windows should be fixed
hover windows now try to size themselves a little better on init, which reduces some initial flicker or false-positive single-frame display on some systems
extended the hover report mode to report some 'ideal' pos/size info as well
under file->shortcuts, custom shortcuts are now hidden for non-advanced-mode users
.
the rest:
the popup message toaster now always shows its 'dismiss all' summary bar whenever any messages are being displayed. the summary bar now also has a ▼/▲ button to collapse/expand its messages!
added duplicate comparison score options (under options->duplicates) for the new jpeg quality estimator
fixed the default duplicate comparison score values, which appeared to be reversed for higher vs much higher--they will be reset to these new defaults on update, so recheck them if you prefer different
in manage tag siblings and parents, the filename tagging dialog, and some misc options panels, tag autocomplete input controls now have 'paste' buttons to make entering many results much easier
to reduce update flicker, the downloader and watcher pages do not list seconds in their 'added' column ('12 minutes 24 seconds ago' is now '12 minutes ago')
improved clipboard access cleanup on in-clipboard errors, which was sometimes leading to error popups or clipboard lockup
rather than the simple 'yes', the review bandwidth usage dialog now puts the waiting estimate (like '12 minutes 50 seconds') in the 'blocked?' column
improved external program launch code for non-Windows to remove hydrus-specific LD_LIBRARY_PATH completely when no OS default exists to restore it to. this should fix ffmpeg connection for certain installs
fixed a rare bug when initial media results of a page failed to load due to a subset of unexpectedly unfetchable file records
gave the rare 'ui freezup on dialog close' event yet another pass. closing via escape key should now be immune to this
'remote' files that were once in the client but since deleted now have the 'trashed' icon and will state so on their right-click info summary lines
fixed various instances where selection-from-list dialogs were failing to sort their list based on underlying data object uncomparibility. an example of this was when selecting which queries to pull from a separating subscription
on the edit parser panels, fetching test data successfully via the quick button or the manual URL entry will now set that URL in the example parsing context
on the edit parser panels, the subsidiary page parser's separation formula now launches with the correct example data (the original data from the parent dialog, rather than the post-separated data) on which to test separation. this should nest correctly for multiple subsidiary page parsers
to reduce server load spikes, clientside petition processing now approves very large mapping and file petitions (such as a petition to delete one tag from 50k files) as a sequence of smaller chunks
moved better/worse/same quality duplicates relationships to the new 'king' group-based model. rather than tracking every relationship, duplicates are now stored in groups with a single 'best' file
as a result, duplicate relationships are now transitive! saying that one king is duplicate to another will merge groups. the 'better' king is the new king, and 'same quality' kings choose one of the kings pseudorandomly. advanced exceptions: saying that a king is better than a basic member of another group or saying that two members are same quality is still valid but will simply 'poach' the non-king member from the other group in order to ensure the wrong king doesn't end up on top in the eventual merge. saying KingA is same quality as MemberB will merge the groups with KingB as the new king (since it is presumably same/better quality to all members of A)
the thumbnail right-click 'duplicates' entry is now renamed to 'file relationships' and is no longer advanced mode only. the 'find similar files' entry is folded into this
the thumbnail 'file relationships' menu now shows a simple 'duplicates' count rather than the old messy better/worse/equal. it will show all the members of a duplicates group when clicked. the menu also notes if the focused file is the best file of its group. if it is not, you will get the option to show the best file or make the focused file the best
as a result, it is now much simpler to view a group of duplicates and overrule a 'best quality' member as needed
added the 'media' shortcut 'duplicate_media_set_focused_king' to shortcut setting a 'best quality' file
the system:num duplicate relationships now has the simpler 'duplicates' entry, to search on size of the entire group. searching for kings/not kings will come soon
due to the new duplicate transitivity rules, potential pairs are now eliminated at a much faster rate!
setting duplicate relationships will overrule false positive or alternate relationships already in place
manually setting alternate relationships to more than two thumbnails at once will now set each file as alternate to every other file in the selection, completely eliminating potential pairs within the group. if you try to do this to large groups of files you will get a longer yes/no confirmation message just to make sure you aren't overwriting some potential dupes by accident
all existing better/worse/same relationships will be converted to the new group storage in this update, with appropriate kings determined. potential pair queue counts will be reduce accordingly, and the temporary alternate/duplicate confusion from the alternates update will be auto-resolved by merging truly duplicate 'alternates' together
fleshed out the duplicate test code significantly to handle the new dupe groups and their interactions with the recent false positive and alternates changes
refactored some db test code into separate client/server/duplicates files and cleaned up dupe tests readability
potential pairs are now the only component of the new system still on the old pairs system. the duplicate filter will still serve up some inefficient (i.e. non-king) comparisons
.
the rest:
fixed the issue where many clipboard-watcher-caught URLs that did not match were producing false-positive 'could not generate new page for that URL' error popups
the clipboard text-fetcher now tests against incompatible clipboard types (like a screenshot) better, and all instances of text fetching now report errors more gracefully and with more information
fixed the unusual OS X issue where many shortcuts were not being processed after client boot until the top menubar was opened and closed. a variety of other blocking-while-menubar-is-open issues that were false-positive misfiring are now fixed as well, please let me know if you still have trouble here
the file menu now has an 'exit and force shutdown maintenance' option to force-run outstanding maintenance jobs
when shutdown maintenance work is going on, the shutdown splash screen now has a 'stop shutdown maintenance' button!
cleaned up some file maintenance manager maintenance locking and shutdown cancel logic
moved all the idle-mode maintenance checks to a new system that explicitly defines idle/shutdown/forced maintenance work and tests those states in a unified manner, checking idle mode and the new splash cancel button status and so on more reliably. a lot of maintenance should cancel out quicker when appropriate
misc shutdown logic cleanup
added a 'file maintenance' option to the database->maintenance menu that forces the new file maintenance manager to run its queue. it'll make a little popup as it works, or a note that no work is due
the 'regenerate' thumbnail menu is also available to all users
jpeg quality estimates are now available for all users in the duplicate filter. they only display when the two jpegs' quality have different labels
the jpeg quality estimator now handles some unusual jpegs that load with empty quantization table arrays
the duplicate filter now handles bad jpeg quality estimations gracefully
cleaned up some ffmpeg communication code
the ffmpeg debug text that spawns on a help->about call that fails to discover ffmpeg version information now prints stderr output as well. if you have been hit by this, please give it another go and let me know what you get
the same ffmpeg 'no response' error on file parse now popups and prints some debug info and returns a better error
dialogs and windows on the new panel system now support a new pre-close tidying system
the manage tags dialog and window will now cancel any pending large tag autocomplete queries on close
regular gui pages now support a new pre-close tidying system
search pages will now cancel any pending search results loading or tag autocomplete queries on close
improved reliability of the popup message manager chasing the main gui when it is sent to another screen by a keyboard shortcut (such as shift+win+arrow on Windows). it should work now if the mouse cursor is in either window. please let me know if this causes trouble for virtual display navigation
the network engine now waits significantly longer--60s--on connection errors before trying again, and with every failed attempt will wait n times longer again. when in this waiting state, a manual user cancel command cancels it out faster
I believe I have fixed/improved a situation where media viewer hover windows would sometimes disappear immediately after appearing on some Linux window managers
improved hover window report mode to state more focus info in case the above is insufficient
to better link the two requests and consume bandwidth under strict rules more precisely, the override bandwidth rule that kicks in when a file page has a single file is now 3 seconds instead of 30
updated options->connection page to specify that 'socks4a'/'socks5h' is needed to force remote dns resolution
sped up tag parents initialisation
repositories now group tag sibling and parent petitions by the parent/better tag's namespace
removed some old network 'death time' code that is no longer useful and was interfering with heavy petition processing
the log now flushes itself to disk every 60s rather than 300s
retuned the new alternate and false positive setting code to be less aggressive about removing potential pairs. users with alternate groups who updated to 354 may have lost some intra-alternate potential pairs, which I expect to fill back in once the potential pairs system is updated
the duplicate filter center-right hover window now has a trash button, which is moved to its own row with the cog icon where they will fit on thinner windows better
users who are in advanced mode now have access to duplicate merge options for 'alternates' again--but be careful with this, it is advanced. if you had merge options for alternates set up before, your old options _should_ return, but make sure to check it
if you are in advanced mode, the duplicate filter can now show an experimental jpeg quality comparison statement. if this works out, I will add custom scores and improve this otherwise based on feedback , so comments would be appreciated
.
client api:
the client api has a new 'manage pages' permission!
/manage_pages/get_pages now fetches a structure of the current page session! there's some help for it
in future, I expect to add a 'select page' command and get/remove/highlight URL(s) for downloader pages, and reveal the unique page identifier for better targeting here and for add_url commands
client api is now version 7
.
the rest:
fixed a stupid issue related to drawing collection thumbnails that was causing a lot of errors. I apologise for the inconvenience and have added pre-build tests to catch these simple mistakes in future
added a clipboard url watcher! there are two options now under network->downloaders--one for watcher urls, and one for all other _recognised_ urls (i.e. post and gallery urls, and file urls that have matching url classes). it checks every second, sends urls to pages according to the same rules as a drag and drop event, and will handle newline-separated lists of urls
when the client is minimised and needs to create a new downloader page because of an 'add url' command through this new clipboard watcher or the client api, it will now queue up the url and its page creation for when the client is next restored (page creation while minimized causes layout bugs). it returns an appropriate result text to the client api in this case
added four 'show_and_focus_manage_tags_XXX' shortcuts to the 'main_gui' shortcut set that let you select and focus the four possible suggested tags panels on the manage tags dialog. these are slightly special focus events that do some 'first tag selection' work as well and should let you make the whole process keyboard-only
added a 'focus_media_viewer' shortcut to the 'main_gui' shortcut set that focuses the media viewer from a media viewer's manage tags window
file viewing statistics gets an update: it now has its own options page where you can set min/max for preview and media viewing times. the global turn on/off control is moved from database menu to this page, as is a new control for enabling it on the duplicate filter (where you will be flicking back and forth and perhaps do not want lots of views recorded). duplicate filter file stats now default to disabled
added a new command to database->file viewing statistics that will cull your current stats based on the new min/max values to compensate for previous accidental '3day 4hour' preview view that snuck in. a yes/no dialog explains how it works before it goes
wrote a 'mixin' class to let my buttons automatically note current shortcut mapping information to the bottom of button tooltips
the duplicate commands and a variety of other media viewer buttons that work in the new shortcut system now report their current shortcuts in their tooltips!
quick-entering 'namespace:' in a search page's tag autocomplete input will now automatically swap in the special 'namespace:*anything*' predicate
cleaned up code around setting a custom temp_dir through the "temp_dir" command, and made it test the given path is indeed a directory and writable-to. if not, the program dumps out with an error popup
the client and server now similarly test that the directory db is a dir and writable-to
rearranged some critical boot error handling code and made the client's boot error handling throw up some ui
fixed the client's repairdb function when needing to regen autocomplete caches
autocomplete regen now publishes status updates to the splash screen
fixed the client's repairdb to wait correctly for the ui in case the user wants to bail before repair
the client's repair db now repairs the new local_tags_cache table if missing
fixed an issue with showing the manage logins panel when the domain had some unusual cookie expiry numbers
to reduce initial load gui-event crunch (which affects some systems' layout reliability), initial session load now happens after a 0.5s delay
cleaned up some image load error handling
when FFMPEG version information cannot be understood in help->about, a popup note appears and debug information is printed to the log
the advanced content update panel now puts up a 'working...' dialog while it processed a job
in the client
all 4XX and 5XX network exceptions' error texts are now prepended with the actual status code
extended the hover window debug report mode
improved some misc temp and permission testing code
duplicates 'false positive' and 'alternates' pairs are now stored in a new more efficient structure that is better suited for larger groups of files
alternate relationships are now implicitly transitive--if A is alternate B and A is alternate C, B is now alternate C
false positive relationships remain correctly non-transitive, but they are now implicitly shared amongst alternates--if A is alternate B and A is false positive with C, B is now false positive with C. and further, if C alt D, then A and B are implicitly fp D as well!
your existing false positive and alternates relationships will be migrated on update. alternates will apply first, so in the case of conflicts due to previous non-excellent filtering workflow, formerly invalid false positives (i.e. false positives between now-transitive alternates) will be discarded. invalid potentials will also be cleared out
attempting to set a 'false positives' or 'alternates' relationship to files that already have a conflicting relation (e.g. setting false positive to two files that already have alternates) now does nothing. in future, this will have graceful failure reporting
the false positive and alternate transitivity clears out potential dupes at a faster rate than previously, speeding up duplicate filter workflow and reducing redundancy on the human end
unfortunately, as potential and better/worse/same pairs have yet to be updated, the system may report that a file has the same alternate as same quality partner. this will be automatically corrected in the coming weeks
when selecting 'view this file's duplicates' from thumbnail right-click, the focus file will now be the first file displayed in the next page
.
duplicates boring details:
setting 'false positive' and 'alternates' status now accounts for the new data storage, and a variety of follow-on assumptions and transitive properties (such as implying other false positive relationships or clearing out potential dupes between two groups of merging alternates) are now dealt with more rigorously (and moreso when I move the true 'duplicate' file relationships over)
fetching file duplicate status counts, file duplicate status hashes, and searching for system:num_dupes now accounts for the new data storage r.e. false positives and alternates
new potential dupes are culled when they conflict with the new transitive alternate and false positive relationships
removed the code that fudges explicit transitive 'false positive' and 'alternate' relationships based on existing same/better/worse pairs when setting new dupe pairs. this temporary gap will be filled back in in the coming weeks (clearing out way more potentials too)
several specific advanced duplicate actions are now cleared out to make way for future streamlining of the filter workflow:
removed the 'duplicate_media_set_false_positive' shortcut, which is an action only appropriate when viewing confirmed potentials through the duplicate filter (or after the ' show random pairs' button)
removed the 'duplicate_media_remove_relationships' shortcut and menu action ('remove x pairs ... from the dupes system'), which will return as multiple more precise and reliable 'dissolve' actions in the coming weeks
removed the 'duplicate_media_reset_to_potential' shortcut and menu action ('send the x pairs ... to be compared in the duplicates filter') as it was always buggy and lead to bloating of the filter queue. it is likely to return as part of the 'dissolve'-style reset commands as above
fixed an issue where hitting 'duplicate_media_set_focused_better' shortcut with no focused thumb would throw an error
started proper unit tests for the duplicates system and filled in the phash search, basic current better/worse, and false positive and alternate components
various incidences of duplicate 'action options' and similar phrasing are now unified to 'metadata merge options'
cleaned up 'unknown/potential' phrasing in duplicate pair code and some related duplicate filter code
cleaned up wording and layout of the thumbnail duplicates menu
.
the rest:
tag blacklists in downloaders' tag import options now apply to the parsed tags both before and after a tag sibling collapse. it uses the combined tag sibling rules, so feedback on how well this works irl would be appreciated
I believe I fixed the annoying issue where a handful of thumbnails would sometimes inexplicitly not fade in after during thumbgrid scrolling (and typically on first thumb load--this problem was aggravated by scroll/thumb-render speed ratio)
when to-be-regenerated thumbnails are taken off the thumbnail waterfall queue due to fast scrolling or page switching, they are now queued up in the new file maintenance system for idle-time work!
the main gui menus will now no longer try to update while they are open! uploading pending tags while lots of new tags are coming in is now much more reliable. let me know if you discover a way to get stuck in this frozen state!
cleaned up some main gui menu regeneration code, reducing the total number of stub objects created and deleted, particularly when the 'pending' menu refreshes its label frequently while uploading many pending tags. should be a bit more stable for some linux flavours
the 'fix siblings and parents' button on manage tags is now a menu button with two options--for fixing according to the 'all services combined' siblings and parents or just for the current panel's service. this overrides the 'apply sibs/parents across all services' options. this will be revisited in future when more complicated sibling application rules are added
the 'hide and anchor mouse' check under 'options->media' is no longer windows-only, if you want to test it, and the previous touchscreen-detecting override (which unhid and unanchored on vigorous movement) is now optional, defaulting to off
greatly reduced typical and max repository pre-processing disk cache time and reworked stop calculations to ensure some work always gets done
fixed an issue with 'show some random dupes' thumbnails not hiding on manual trashing, if that option is set. 'show some random dupes' thumbnail panels will now inherit their file service from the current duplicate search domain
repository processing will now never run for more than an hour at once. this mitigates some edge-case disastrous ui-hanging outcomes and generally gives a chance for hydrus-level jobs like subscriptions and even other programs like defraggers to run even when there is a gigantic backlog of processing to do
added yet another CORS header to improve Client API CORS compatibility, and fixed an overauthentication problem
setting a blank string on the new local booru external port override option will now forego the host:port colon in the resultant external url. a tooltip on the control repeats this
reworded and coloured the pause/play sync button in review services repository panel to be more clear about current paused status
fixed a problem when closing the gui when the popup message manager is already closed by clever OS-specific means
duplicate action options no longer handle file deletion
renamed 'not duplicates' across the program to 'not related' or 'false positive'
'alternates' and 'not related/false positive' duplicate actions no longer have duplicate action options. no merge content update now occurs on these actions
the duplicate filter hover panel now splits 'this is better' decisions into two buttons--whether to delete or keep the worse file
when selecting 'custom action' in the duplicate filter hover panel, it now asks if you would like to delete the current file, the other file, or both
the 'duplicate_filter_this_is_better' shortcut action will be auto-updated to 'duplicate_filter_this_is_better_and_delete_other'. an alternate 'duplicate_filter_this_is_better_but_keep_both' is now also available
the 'duplicate_filter_not_dupes' shortcut action will be auto-updated to 'duplicate_filter_false_positive'
separated the buttons on the duplicate filter hover panel to more carefully split 'yes, files are duplicates' vs other decisions
in prep for the duplicate db overhaul, refactored all PHash search code and Duplicate management code apart
misc other prep work for duplicate db overhaul
.
file maintenance:
wrote a new unified manager to handle various long-term file maintenance tasks like regenerating file metadata and thumbnails
options to govern how this manager can run are now in options->maintenance and processing. you can enable it for idle and shutdown maintenance time and give it a throttle to limit how fast it will work on files, defaulting to 200 per day
unified the previous db-level attempts at file maintenance to the new system, which supports async job queueing, and moving regen code up to the new manager, out of the db lock
unified a variety of file and thumbnail regen code to work through the new simpler and saner path
the right-click->regen thumbnail commands now run through the new manager and no longer need a modal popup. you can keep browsing while they work. they will also not hang the ui as the old system could on big jobs
when right-click->regenning on more than 50 thumbnails, you now get a dialog asking if you want to do the job now or put it off later
file maintenance tasks can now run in shutdown time! you will get previews of the jobs with file counts and status progress reports on the shutdown splash
cleaned up some file extension renaming and dupe-removing code
in future, I will move the current file integrity check to this new system and have some ui to prompt and set up other big jobs, like fixing various historical misparsing issues
thumbnail resizing during thumbnail fade that resizes down is now more efficient
moved the ClientFilesManager to ClientFiles.py
.
the rest:
the 'manage upnp' dialog now moves the duplicated external ip display from the column up to the status text at the top. it fetches the ip after the initial mappings fetch is done. this ip is no longer affected by the external host override option
cleaned up options->connection page and removed the now defunct external host override option
the manage services page for the local booru now has optional override for scheme, host, and port for the 'copy external url' function
fixed an issue with the recent 'collect by' session saving where a restored session that needed a collect was not sorted
fixed an issue with collections being sorted by approx bitrate
added a new checkbox to options->sort/collect to set it so the default sort updates every time you choose a new sort anywhere
fixed an issue with 'remove trashed files from view', which was incorrectly removing on 'all local files' pages
the 'all local files' file domain, which is frequently confusing to new users, is now no longer an option for new file pages or the autocomplete file domain if the user is not in advanced mode
the client now searches for versions of urls both with and without a final '/' character when looking up file url import status at the db level and in import lists. system:known_url is unfortunately still an inefficient mess
improved how the server code deals with some connectionLost errors
cleaned up and unified some older dialog button code
fixed a problem in manage tag siblings when petitioning existing pairs and then cancelling when asked for a reason
fixed a miscount issue when uploading pending tags while many new tags are coming in. progress would sometimes be -754/1,234, ha ha
db maintenance, repository sync, and file maintenance processing will all now wake on a force idle mode call
the client now supports importing .ico files! (.cur should be supported too)
finally, 'collect by' is saved for sessions! if your default collect by previously included ratings services, it will forget them this one time--please reset it under the options->sort/collect
fixed the issue where the media viewer's hover windows were hovering over child dialogs (manage tags, ratings, or known urls)
improved some os x hover window focus handling for the new always-on-top duplicate action window
the entries on the 'sort by' list on gui pages are now subcategorised better. it should be a bit easier to find what you are looking for
the 'sort by file: approximate bitrate' sort option now sorts still images as well by filesize / num_pixels
to reduce confusion, sort by mime and system:mime are now renamed to 'filetype'
fixed an issue where the 'unclose_page' shortcut was restoring pages in reverse order (unclosing least-recently-closed-first rather than most-recently-closed-first)
improved rigour of video framerate estimation
stopped the video metadata parser from opting to manually frame count videos with size >128MB or num_frames estimate >2,400
fixed the forced manual frame count to deal with frame counts >9999
the 'ffmpeg not found' error on file import will now put up a popup message once per boot informing you of this problem more broadly and steps to address it
fixed some underreporting issues with subprocess_report_mode
fixed an issue with some yes/no dialogs returning 'no' on escape/window_close_button rather than 'cancel', which affected cancelability some db maintenance questions
fixed an issue where media that fitted the media viewer canvas width or height exactly at 100% zoom would not respond to zoom switch events to restore non-100% zoom to 100%
when a local server's CORS mode is turned on, Access-Control-Allow-Origin is now correctly added to GET/POST requests with an Origin request header
improved reliability of some timestamp rendering code, which should help some users who had trouble opening cookies management page after malformed cookie import
I believe I fixed an issue with 'open externally' on certain custom paths where the external program could spawn without an ui (flash projector did this). please let me know if your 'open externally' calls start making terminal windows everywhere
fixed a runtime stability issue with the new duplicates page and slow-updating counts that come in after the page has been deleted
wrote a new (always on top!) hover window for the duplicate filter that sits on the middle-right. the duplicate cog button and action buttons are moved to this new window, as are the file comparison statements
the duplicate file comparison statements now state the relevant actual metadata along with better '>>'-style operators to highlight differences and green/red/blue colouring based on given score. it is now much easier to see and action clearly better files at-a-glance
improved some hover window focus display calculations to play with the new always-on-top tech
both the 'show some random dupes' button and finding dupe pairs for the filter should be a bit faster for very large search domains. the basic file search and indexing still has to run, but the second sampling step in both cases will bail out earlier once it has a decent result
core image handling functions now uniformly use OpenCV (faster, more accurate) by default, falling to PIL/Pillow on errors. image importing in the client and server should be a bit faster, and some unusual image rotations should now be read correctly
the server now supports OpenCV for image operations, it _should_ also still work with only PIL/Pillow, if you are running from source and cannot get it
unified all thumbnail generation code and insulated it from suprises due to unexpectedly-sized source files, fixing a potential client-level thumbnail generation looping bug
gave all image processing a refactor and general cleanup pass, deleted a bunch of old code
wrote a new 'local tag cache' for the db that will speed up tag definition lookups for all local files. this should speed up a variety of tag and file result fetching, particularly right after client boot. it will take a minute or two on update to generate
sped up how fast the tag parent structure builds itself
the review services panel now uses nested notebooks, rather than the old badly coded listbook control. I don't really like how it looks, but the code is now saner
similar-files metadata generation now discards blank frames more reliably
subscription popups now report x/y progress in terms of the current job, discarding historical work previously done. 1001/1003 is gone, 1/3 is in
made the disk cache more conservative on non-pre-processing calls
cleaned up some file import code, moving responsibility from the file locations manager to the file import object
updated the ipfs service listctrl to use the new listctrl object. also cleaned up its action code to be more async and stable
I believe I fixed a rare vector for the 'tryendmodal' dialog bug
fixed a bug in presenting the available importable downloader objects in the easy drag-and-drop downloader import when the multiple downloaders dropped included objects of the same type and name--duplicate-named objects in this case will now be discarded
unified url_match/url_class code differences to url class everywhere
updated some common db list selection code to use new python string formatting
the duplicate filter no longer applies the implicit system limit (which defaults to 10,000 files) on its search domains, which solves the undercounting issue on large search domains. duplicate operations will be appropriately slower, so narrow your duplicate file queries (adding a creator: tag works great) if they take too long
fixed the duplicate pairs filter's 'ghost pair' issue. it was failing, when 'both files' was unchecked, to remove pairs that included one file that was non-local. this accidental inclusion resulted in incorrect non-zero count and filter/random pairs that could not display correctly
insulated against potential future iterations of this problem (likely that one of the pair was deleted by another process while a filter is ongoing), with a notification and graceful exiting of the duplicate filter while saving progress
the 'show random duplicates' button now puts the 'base' of the group (to which all the others are potentially matched) as the first thumbnail
added a new 'advanced file deletion' section to 'files and trash' options page to turn on a new advanced dialog and set custom file deletion reasons
if this new dialog is turned on, a delete event from thumbnail grid, regular media viewer, or the duplicate filter's manual delete will launch it. it permits you to delete physically (skipping trash) in one step or delete physically without leaving a deletion record (for easier later re-import) and choose one of the deletion reasons in the new options panel or set a one-time custom reason
export folders now have more run-controls: 'run regularly', 'paused', and 'run now'
the file menu now has a 'run export folder now' submenu just like for import folders-- it is simple now to set up an export folder that only runs when you tell it to
updated the on-boot missing file folder recovery process to automatically 'heal' file location mappings when a missing folder is actually in a location that is known (essentially, you can now manually move a bunch of folders from one known location to another while the client is off and it will recover automatically now). error dialogs will appear in this case summarising the problem and proposed fixes with a chance to bail out if you do not want it to happen
added a new frame type to 'gui' options page called 'regular_center_dialog' for yes/no style dialogs that are better in the center of the parent window
the custom web browser launch path and file type 'open externally' paths are moved from 'files and trash' to a new 'external programs' options page
as the superior '--temp_path' program launch parameter now exists for both client and server, I have removed the limited 'BUGFIX: temp folder override' option from the client's 'files and trash' page and use in the actual code. if this option was important to you, please migrate to the --temp_path launch parameter, which covers temp usage more comprehensively
as the artstation downloader is now non-functional, apparently by a cloudflare issue, the default gug for new users (and anyone with artstation set atm) is now safebooru
added dolphin file manager add-on link to the client api help
No Comments