Optimizes image files by converting them to webp while also updating all references.

Overview

About

Optimizes images by (re-)saving them as webp. For every file it replaced it automatically updates all references. Works on single files as well as on folders.

Installation: Windows

Step 1: Get the exe

Option 1: Download

Download one of the pre-built fvttoptimizer.exe files from Releases

Go to Step 2: Install program

Option 2: Build it yourself

Install python3 (version >= 3.8) and add it to your PATH system environment variable.

You can check the version via CMD or powershell with:

python --version

Download or clone the repo.

Open a CMD instance inside the project root folder and create a virtual environment.

python -m venv .\venv

Activate the venv:

.\venv\Scripts\activate

Install the Pillow package:

pip install pillow==9.1.0

Install pyInstaller package:

pip install pyInstaller==4.10

Download the fvttmv python package (whl file) from here: https://github.com/watermelonwolverine/fvttmv/releases/tag/0.2.4 move it into the folder and install it:

pip install fvttmv-0.2.4-py3-none-any.whl

Run build file:

.\scripts\build_for_windows.cmd

You should now have a fvttoptimizer.exe file under dist. After the build succeeded you can delete the venv you previously created.

Go to Step 2: Install program

Step 2: Install program

Create an empty folder where you want to install the program for example C:\fvttoptimizer

Copy fvttoptimizer.exe into that folder.

Create a fvttoptimizer.conf text file in that folder.

Copy {"absolute_path_to_foundry_data":"INSERT_PATH_HERE"} into fvttoptimizer.conf

Replace INSERT_PATH_HERE with the path to the Data folder inside your foundrydata (Not the foundrydata folder itself!).

IMPORTANT: Escape all \ with \\ in that path.

It should look something like this:

{"absolute_path_to_foundry_data":"C:\\Users\\user\\foundrydata\\Data"}

Add the installation path to your PATH system environment variable.

Uninstallation: Windows

Delete fvttoptimizer.exe and fvttoptimizer.conf files from the installation directory.

Remove the path to the installation directory from the PATH system environment variable.

Installation: Ubuntu 16.04 -20.04

Optional: Install Python

An up-to-date Ubuntu 20.04 should have python >= 3.8. Check it with:

python3 --version

From here on pythonX will be used as placeholder for the python you should use. Depending on your system you need to replace X with 3, 3.8, 3.9 or 3.10 .

Install python version>=3.8 if not yet installed.

sudo apt install pythonX

Step 1: Get the executable

Option 1: Download

Download one of the pre-built fvttoptimizer files from Releases

Go to Step 2: Install the files

Option 2: Build it yourself

Install python if haven't already.

Install Pillow dependencies (read more here https://pillow.readthedocs.io/en/stable/installation.html):

sudo apt install pythonX-dev libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python3-tk libharfbuzz-dev libfribidi-dev libxcb1-dev

Download or clone repo.

Create venv inside the repo:

sudo apt install pythonX-venv

pythonX -m venv ./venv

Activate venv:

source venv/bin/activate

Install Pillow (should succeed if you installed all the dependencies):

pythonX -m pip install Pillow==9.1.0

Install pyInstaller:

pythonX -m pip install pyInstaller==4.10

Download the fvttmv python package (whl file) from here: https://github.com/watermelonwolverine/fvttmv/releases/tag/0.2.4 move it into the folder and install it:

pythonX -m pip install fvttmv-0.2.4-py3-none-any.whl

Now run the build script:

./scrips/build_for_ubuntu.sh

Step 2: Install the files

Option 1: Automatic installation

Install python if you haven't already.

Clone the repo if you haven't already.

Go into the project folder.

Build the project if you haven't already. If you downloaded a pre-built executable. Create a folder named dist and move the file there.

Inside the project folder run:

sudo pythonX scripts/install_on_ubuntu.sh

Option 2: Manual installation

Copy the fvttoptimizer file either from dist from where you downloaded it to usr/bin/fvttoptimizer

Make the file executable:

sudo chmod ugo=rx usr/bin/fvttoptimizer

Create a fvttoptimizer.conf file at /etc/

Copy {"absolute_path_to_foundry_data":"INSERT_PATH_HERE"} into fvttoptimizer.conf

Replace INSERT_PATH_HERE with the path to the Data folder inside your foundrydata (Not the foundrydata folder itself!).

It should look something like this:

{"absolute_path_to_foundry_data":"/home/user/foundrydata/Data"}

Uninstallation: Ubuntu

Delete the files /etc/fvttoptimizer.conf and /usr/bin/fvttoptimizer

Usage

IMPORTANT:

Shut down Foundry VTT before optimizing any files.

Use this program at your own risk. It has been tested extensively but there are no guarantees. Always keep backup of your Foundry VTT data for cases where something goes wrong.

Syntax

fvttoptimizer [--verbose-info, --verbose-debug, --version, --help, --quality value, --override-percent value, --skip-webp, --skip-existing, --recursive] target

target: Path to file or directory which should be optimized

Options

--verbose-info: Enables verbose output to console

--verbose-debug: Enables very verbose output to console

--version: Prints version and exits

--help: Display help and exit

--skip-existing: Ignores files of which a webp already exists. For example image.png will not be converted if a image.webp is already in the same folder

--skip-webp: Don't touch webp files at all

--quality: The quality setting for the webp compression. Default is 75

--override-percent: How much smaller the new file needs to be to replace the old one. For example if this value is 25 the file size after optimizing needs to be 25% smaller than the original. Default is 25

--recursive: If the optimization should be done recursively to all sub folders of the target folder.

Examples

Optimizing a file:

Ubuntu: fvttoptimizer path/to/file.png

Windows: fvttoptimizer path\to\file.png

Optimizing a folder and all it's sub folders:

Ubuntu: fvttoptimizer --recursive pathto/folder

Windows: fvttoptimizer.exe --recursive path\to\folder

Use single quotes when moving a file with a space:

Ubuntu: fvttoptimizer 'some folder/some file.jpg'

Windows: fvttoptimizer.exe 'some folder\some file.jpg'

Change parameters:

Ubuntu: fvttoptimizer --quality 50 --override-percent 50 file.jpg

Windows: fvttoptimizer --quality 50 --override-percent 50 file.jpg

Known Issues and Quirks

Windows

The program only works in powershell not in cmd.

When one of the paths has \' at the end, the arguments will get mixed up. This is a problem with how python handles arguments and probably can't be fixed. For example on Windows fvttoptimizer.exe '\folder name with spaces\' will fail but fvttoptimizer.exe '\folder name with spaces' will succeed.

Comments
  • Program fails on special character [ in path

    Program fails on special character [ in path

    First off, thank you so much for making this.

    I am trying to use the following command: fvttoptimizer.exe --recursive 'C:\Users\cnguy\AppData\Local\FoundryVTT\Data\Battle Maps'

    and I am getting this return: An unexpected error occurred: expected str, bytes or os.PathLike object, not NoneType

    Your documentation implies this is supported, as this command works without the "--recursive" flag, so maybe there is an issue with spaces in paths and the recursive function?

    Thank you.

    opened by coboba123 5
  • Trying to create a Docker image - Keep getting 'image.jpg is not in the configured foundry Data folder.'

    Trying to create a Docker image - Keep getting 'image.jpg is not in the configured foundry Data folder.'

    As the title say, I'm trying to get this tool to work in a Docker container, but I'm having an issue that I can't figure out...

    I'm building the image like so:

    FROM python:3.10.0
    
    ARG FVTT_OPTIMIZER_VERSION=0.1.3
    
    RUN curl -L https://github.com/watermelonwolverine/fvttoptimizer/releases/download/$FVTT_OPTIMIZER_VERSION/fvttoptimizer --output usr/bin/fvttoptimizer
    RUN chmod +x usr/bin/fvttoptimizer
    RUN echo '{"absolute_path_to_foundry_data":"/app/Data"}' > /etc/fvttoptimizer.conf
    
    CMD ["fvttoptimizer"]
    

    Which I then configure with docker-compose:

    version: "3.7"
    
    services:
        fvttoptimizer:
            build: python
            volumes:
                - ${FOUNDRY_DATA_VOLUME}:/app
    
    # FOUNDRY_DATA_VOLUME is the directory that contains `Config`, `Data` and `Logs`
    

    However, when ran (docker-compose run --rm fvttoptimizer fvttoptimizer --verbose-debug ./Barney.jpg) I get the error in the title. Running docker-compose run --rm fvttoptimizer ls -la /app/Data, I can confirm the image exists.

    The debug output looks like this:

    root - DEBUG - Got arguments ['fvttoptimizer', '--verbose-debug', '--recursive', './Barney.jpg']
    root - DEBUG - Reading config from: /etc/fvttoptimizer.conf
    root - DEBUG - Running with target_path='./Barney.jpg', config='{'_RunConfig__program_config': {'_ProgramConfigImpl__abs_path_to_foundry_data': '/app/Data'}, 'skip_existing': False, 'skip_webp': False, 'recursive': True, 'quality': 75, 'override_percent': 25}'
    root - ERROR - /Barney.jpg is not in the configured foundry Data folder.
    /Barney.jpg is not in the configured foundry Data folder.
    

    I pushed a branch on my repo where it should be possible to reproduce the issue, or at the very least look at my configuration: https://github.com/gCardinal/foundrty-vtt/blob/feature/use-fvttoptimizer

    support 
    opened by gCardinal 3
  • Missing config file

    Missing config file

    When running on an image file I get the error:

    Missing config file. Could not find /etc/fvttoptimizer.conf

    Tried running all three release versions on Arch Linux with Python 3.10.4 and get the same error every time.

    I'm not seeing any mention of a config file in the readme. Am I missing something?

    support 
    opened by tonydero 2
  • Suggestion: Print which file it is working on during the conversion process.

    Suggestion: Print which file it is working on during the conversion process.

    I think it would be nice to have some kind of progress indicator to show what its working on in the command prompt window. The easiest way to have this would simply just print the name of the file that is being worked on when it gets to it. When working on folders with lots of images, its nice to know its not frozen or hanging on something.

    Thank you!

    opened by coboba123 2
  • getting root - ERROR - [path] is not in the configured foundry Data folder.

    getting root - ERROR - [path] is not in the configured foundry Data folder.

    So I am running on a Google Cloud server, and have manually installed fvttoptimizer with Python 3.8.

    I created the /etc/fvttoptimizer.conf with the following line:

    {"absolute_path_to_foundry_data":"/home/tehjoester/foundrydata/Data"}

    and here is my output with the --verbose-debug option:

    [email protected]:~$ sudo fvttoptimizer --verbose-debug --skip-webp --recursive 'foundrydata/Data/'
    root - DEBUG - Got arguments ['fvttoptimizer', '--verbose-debug', '--skip-webp', '--recursive', 'foundrydata/Data/']
    root - DEBUG - Reading config from: /etc/fvttoptimizer.conf
    root - DEBUG - Running with target_path='foundrydata/Data/', config='{'_RunConfig__program_config': {'_ProgramConfigImpl__abs_path_to_foundry_data': '/home/tehjoester/foundrydata/Data'}, 'skip_existing': False, 'skip_webp': True, 'recursive': True, 'quality': 75, 'override_percent': 25}'
    root - ERROR - /home/tehjoester/foundrydata/Data is not in the configured foundry Data folder.
    /home/tehjoester/foundrydata/Data is not in the configured foundry Data folder.
    
    opened by thejoester 1
  • Argument --skip-webp does not seem to work

    Argument --skip-webp does not seem to work

    It's entirely possible that I am doing something wrong, but when I try to use --skip-webp it still tries to process webp files. Running the optimizer twice in a row shows that it will attempt the same files again even if they've already been converted.

    I am running Windows 11. This is the command I tried to run in Windows Terminal/PowerShell. Variations on this command have not resolved the issue.

    fvttoptimizer.exe --verbose-info --skip-webp --quality 100 --override-percent 10 --recursive 'C:\Games\Foundry Virtual Tabletop Data\Data\worlds\mitera'

    opened by seanms1991 0
  • Some actors are not getting the directory to their new avatar image updated properly

    Some actors are not getting the directory to their new avatar image updated properly

    While most actors have their pictures updated as expected, some are skipped. From what I can tell, it's mostly the avatars of the actors and not the tokens. The images are updated but the reference is not updated in the actors.db. Attached are some examples of actors that have this issue. Their tokens were created with Tokenizer, but those converted successfully and their main picture did not, which was not created with tokenizer. I've uploaded the pictures that were converted, the actor data exported as json files, and the actors.db files they were in both before and after the optimizer processed it.

    I am running Windows 11 and have tried to run this on two separate computers with the same result.

    If you need more information, let me know.

    Example.zip

    opened by seanms1991 0
  • Feature Request

    Feature Request

    First off, this script is amazing. It's greatly reduced the sizes of my images directories.

    It would be really cool though if I could maybe put in the config file the path to personal modules that it will update the file names. I have several shared compendiums in my own module that I move between worlds.

    enhancement 
    opened by Jonwh25 1
  • Program fails when a db file contains a byte UTF-8 can't decode.

    Program fails when a db file contains a byte UTF-8 can't decode.

    I got the following message in powershell:

    `PS C:\FVTTOptimizer> .\fvttoptimizer.exe --verbose-info --skip-webp --recursive "C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja" root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map' root - INFO - Optimizing C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Animal Market Stall.png. Decreased file size by 83% root - ERROR - Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in __maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in __optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

    An internal error occurred: Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in __maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in __optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

    Please file a bug report on https://github.com/watermelonwolverine/fvttoptimizer/issues`

    bug 
    opened by YourDNDPleasesMe 10
Releases(0.1.3)
Owner
Watermelon Wolverine
Watermelon Wolverine
Face recognition system using MTCNN, FACENET, SVM and FAST API to track participants of Big Brother Brasil in real time.

BBB Face Recognizer Face recognition system using MTCNN, FACENET, SVM and FAST API to track participants of Big Brother Brasil in real time. Instalati

Rafael Azevedo 232 Dec 24, 2022
LoFTR:Detector-Free Local Feature Matching with Transformers CVPR 2021

LoFTR-with-train-script LoFTR:Detector-Free Local Feature Matching with Transformers CVPR 2021 (with train script --- unofficial ---). About Megadepth

Nan Xiaohu 15 Nov 04, 2022
Plenoxels: Radiance Fields without Neural Networks

Plenoxels: Radiance Fields without Neural Networks Alex Yu*, Sara Fridovich-Keil*, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa UC Be

Sara Fridovich-Keil 81 Dec 25, 2022
Official Pytorch implementation of RePOSE (ICCV2021)

RePOSE: Iterative Rendering and Refinement for 6D Object Detection (ICCV2021) [Link] Abstract We present RePOSE, a fast iterative refinement method fo

Shun Iwase 68 Nov 15, 2022
Opinionated code formatter, just like Python's black code formatter but for Beancount

beancount-black Opinionated code formatter, just like Python's black code formatter but for Beancount Try it out online here Features MIT licensed - b

Launch Platform 16 Oct 11, 2022
Incorporating Transformer and LSTM to Kalman Filter with EM algorithm

Deep learning based state estimation: incorporating Transformer and LSTM to Kalman Filter with EM algorithm Overview Kalman Filter requires the true p

zshicode 57 Dec 27, 2022
This code is a toolbox that uses Torch library for training and evaluating the ERFNet architecture for semantic segmentation.

ERFNet This code is a toolbox that uses Torch library for training and evaluating the ERFNet architecture for semantic segmentation. NEW!! New PyTorch

Edu 104 Jan 05, 2023
This repo provides the base code for pytorch-lightning and weight and biases simultaneous integration.

Write your model faster with pytorch-lightning-wadb-code-backbone This repository provides the base code for pytorch-lightning and weight and biases s

9 Mar 29, 2022
N-Omniglot is a large neuromorphic few-shot learning dataset

N-Omniglot [Paper] || [Dataset] N-Omniglot is a large neuromorphic few-shot learning dataset. It reconstructs strokes of Omniglot as videos and uses D

11 Dec 05, 2022
Official code for "Stereo Waterdrop Removal with Row-wise Dilated Attention (IROS2021)"

Stereo-Waterdrop-Removal-with-Row-wise-Dilated-Attention This repository includes official codes for "Stereo Waterdrop Removal with Row-wise Dilated A

29 Oct 01, 2022
FSL-Mate: A collection of resources for few-shot learning (FSL).

FSL-Mate is a collection of resources for few-shot learning (FSL). In particular, FSL-Mate currently contains FewShotPapers: a paper list which tracks

Yaqing Wang 1.5k Jan 08, 2023
High accurate tool for automatic faces detection with landmarks

faces_detanator High accurate tool for automatic faces detection with landmarks. The library is based on public detectors with high accuracy (TinaFace

Ihar 7 May 10, 2022
Deep Learning to Improve Breast Cancer Detection on Screening Mammography

Shield: This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Deep Learning to Improve Breast

Li Shen 305 Jan 03, 2023
Neural Tangent Generalization Attacks (NTGA)

Neural Tangent Generalization Attacks (NTGA) ICML 2021 Video | Paper | Quickstart | Results | Unlearnable Datasets | Competitions | Citation Overview

Chia-Hung Yuan 34 Nov 25, 2022
A criticism of a recent paper on buggy image downsampling methods in popular image processing and deep learning libraries.

A criticism of a recent paper on buggy image downsampling methods in popular image processing and deep learning libraries.

70 Jul 12, 2022
This repo is for segmentation of T2 hyp regions in gliomas.

T2-Hyp-Segmentor This repo is for segmentation of T2 hyp regions in gliomas. By downloading the model from here you can use it to segment your T2w ima

1 Jan 18, 2022
Active and Sample-Efficient Model Evaluation

Active Testing: Sample-Efficient Model Evaluation Hi, good to see you here! 👋 This is code for "Active Testing: Sample-Efficient Model Evaluation". P

Jannik Kossen 19 Oct 30, 2022
Bottleneck Transformers for Visual Recognition

Bottleneck Transformers for Visual Recognition Experiments Model Params (M) Acc (%) ResNet50 baseline (ref) 23.5M 93.62 BoTNet-50 18.8M 95.11% BoTNet-

Myeongjun Kim 236 Jan 03, 2023
CS583: Deep Learning

CS583: Deep Learning

Shusen Wang 2.6k Dec 30, 2022
Run PowerShell command without invoking powershell.exe

PowerLessShell PowerLessShell rely on MSBuild.exe to remotely execute PowerShell scripts and commands without spawning powershell.exe. You can also ex

Mr.Un1k0d3r 1.2k Jan 03, 2023