Pueue data migration from '0.4.0' to '0.5.0' versions

pueue from 0.4.0 To 0.5.0 Version data migration

pueue from 0.3.0 Upgrade to 0.4.0	pueue from 0.4.0 Upgrade to 0.5.0
https://huzhenghui.blog.csdn.net/article/details/106004623	this paper

About pueue

pueue Is a command line task management tool ,pueue It's not like ps Management tools for all tasks of the management system , It's a tool for scheduling and executing long-running tasks sequentially and in parallel ,pueue You can manage the execution sequence of these tasks , And track the status and output of these long-running tasks .

In short ,pueue It's a shell Command queue management tool , name pueue That is, from queue queue , what's more , There are many convenience functions and abstract management functions .

pueue Run as a daemons , Therefore, it is not bound to any terminal , Therefore, any terminal on the same computer can control tasks . Even close all ssh conversation , The queue is still processed continuously .

pueue The main functions include ：

• When planning the command to be executed, you can specify the working directory of the command execution . adopt pueue status You can view the working directory of each command , stay 0.5.0 You can also get the working directory in the callback command .
• You can easily view the output of the command . adopt pueue log You can view it in the form of log . stay 0.4.0 And previous versions can be used pueue show Check the output of the running command , stay 0.5.0 Version changed to pueue follow Check the output of the running command . pueue follow The usage of this function is detailed in the following chapters
• It can interact with running processes
• When it is necessary to release the processing power of the computer , Can pause at any time / Recovery task
• It is convenient to operate the arranged task sequence
• You can run multiple tasks in parallel at the same time , You can also specify the number of tasks you want to run in parallel
• since 0.5.0 Start , You can group tasks . Each group can be regarded as an independent task queue , And it can run multiple tasks in parallel . This is also one of the necessary data migration involved in this article , See the following content for detailed task grouping usage
• since 0.5.0 Start , Support callback hook . for example , Set up desktop notifications . See the following content for detailed task grouping usage
• All functions support Linux and MacOS , Windows Some functions are supported on

pueue0.5.0 Version Description

pueue On 2020 year 5 month 16 Promulgated by the 0.5.0 edition , Include ：

New characteristics

• grouping . Now you can assign tasks to groups . Each group can be regarded as an independent task queue , And it can run multiple tasks in parallel . This is also one of the necessary data migration involved in this article , See the following content for detailed task grouping usage

• Users can specify custom callbacks . Call the callback function when the task ends . See the following content for detailed task grouping usage

change

• It can be used during operation and pause log , Before, you can only use show , So this change makes log Replaced the show Part of the function of .

• Corresponding , show Change it to follow . at present follow Only used to track the output of a single command . pueue follow The usage of this function is detailed in the following chapters

improvement

• Capture environment variables . When the task starts, it will use the call pueue add The environment variable of time , Instead of the environment variable when the task is started . This is also one of the necessary data migration involved in this article
• follow （ It is the former show ） stay read_local_logs Set to true when , Also read directly from disk .
• --all The mark is used for kill 、 start and pause Affect all groups and default groups .
• status increase --group Mark , Used to output tasks of the specified group . The usage of this function is detailed in the following chapters
• kill increase --default , Using this tag will only affect tasks in the default queue .

pueue0.5.0 Data migration

although pueue0.5.0 The release notes do not directly address the issue of data migration , But in the 0.4.0 Upgrade to 0.5.0 After that, you need to migrate data to run normally . The status file to be migrated is in

~/.local/share/pueue/state.json

Stop the daemon first

First stop pueue

pueue shutdown

After complete shutdown, the following message will be displayed

Daemon is shutting down

Backup pueue Status file

Back up with the current version number pueue Status file

jq '' ~/.local/share/pueue/state.json | tee ~/.local/share/pueue/state.$(pueue -V | grep -o '[0-9.]\+').json

Add a group of daemons groups Field

function pueue.

pueued -vvv

If you see the following error

[ERROR] Failed to deserialize previous state log: Error("missing field `groups`", line: 15, column: 5)

Description: the daemon settings in the status file are missing grouping groups Field . This field is used for the above grouping function . The usage of this function is detailed in the following chapters

The data migration method is

jq '.settings.daemon.groups={}' ~/.local/share/pueue/state.0.4.0.json > ~/.local/share/pueue/state.json

The usage of this function is detailed in the following chapters

Add environment variables for each task envs Field

function pueue.

pueued -vvv

If you see the following error

[ERROR] Failed to deserialize previous state log: Error("missing field `envs`", line: 34, column: 5)

Description each task in the status file is missing an environment variable envs Field . This field is used for the aforementioned capture environment variable function , The environment variable set here is empty , You can set as needed when adding .

Add the data migration method based on the previous script as

jq '.settings.daemon.groups={}' ~/.local/share/pueue/state.0.4.0.json |
	jq '.|(.tasks[].envs={})' > ~/.local/share/pueue/state.json

Add groups in the status file groups Field

function pueue.

pueued -vvv

If you see the following error

[ERROR] Failed to deserialize previous state log: Error("missing field `groups`", line: 9211, column: 1)

Indicates the lack of grouping in the status file groups Field . This field is used for the above grouping function .

Add the data migration method based on the previous script as

jq '.settings.daemon.groups={}' ~/.local/share/pueue/state.0.4.0.json |
	jq '.|(.tasks[].envs={})' |
	jq '.groups={}' > ~/.local/share/pueue/state.json

The usage of this function is detailed in the following chapters

pueue0.5.0 New callback command function

The callback command is 0.5.0 New features , You need to modify the configuration file directly , The location of configuration files is different on different operating systems ,MacOS The configuration file is located on ：

~"/Library/Preferences/pueue.yml"

Back up before modifying .

yq read ~"/Library/Preferences/pueue.yml" | tee ~"/Library/Preferences/pueue.$(date).yml"

Set callback command .

yq write --inplace ~"/Library/Preferences/pueue.yml" 'daemon.callback' "export command=\$(echo {
      {
       command }}); osascript -e 'display notification (system attribute \"command\") with title \"Pueue {
    {group}} : {
    { id }} : {
    { result }}\" subtitle \"{
    { path }}\"'"

The command is MacOS A notice is displayed on , Set on demand on different operating systems , The location of the callback command in the configuration file is daemon.callback, When setting the command line, you need to escape the nested quotation marks , Therefore, double quotation marks should be used for this parameter , After this setting, the contents in the view configuration file are the same .

"export command=\$(echo {
   { command }}); osascript -e 'display notification (system attribute \"command\") with title \"Pueue {
   {group}} : {
   { id }} : {
   { result }}\" subtitle \"{
   { path }}\"'"

The command to remove escape is ：

export command=$(echo {
     {
      command }});
osascript -e 'display notification (system attribute "command") with title "Pueue {
    {group}} : {
    { id }} : {
    { result }}" subtitle "{
    { path }}"'

among { { command }} Parameters are not used directly as other parameters osascript Script , Instead, save it in the environment variable first .

export command=$(echo {
     {
      command }});

This is because { { command }} It may contain various possible contents , Directly splicing strings may lead to syntax errors caused by escape characters .

The second line osascript The command is to execute OSA Script , Parameters -e Indicates that the next string is the script to be executed , The script content is split into multiple lines （ Just for the convenience of reading , Cannot perform ） by ：

display notification (system attribute "command")
  with title "Pueue {
   {group}} : {
   { id }} : {
   { result }}"
  subtitle "{
   { path }}"

among ：

• display notification Indicates display notification , The parameter on the right is the content of the notification ,(system attribute "command") Means to get... From the environment variable command Value , Passing strings with variables helps to reduce the possibility that concatenated strings may cause syntax errors caused by escape characters
• with title Indicates setting the title , The parameter on the right is the title of the notification
• subtitle Indicates setting subtitles , The parameter on the right is the subtitle of the notification

Nested in the content string pueue Parameters of callback command , This example uses all known parameters .

Parameters	meaning
{ { id }}	The sequence number of the task
{ { command }}	The command line of the task . When using, you need to pay special attention to the syntax errors caused by escape characters when directly splicing strings
{ { path }}	The path of the task
{ { result }}	Results of the task
{ { group }}	Task group

Again , This splicing string may splice unexpected commands when some parameters contain escape characters , For more complex processing, only id, And then use pueue status --json Query all the States and parse out the required commands . The following examples give the methods to solve common escape conflicts

pueue0.5.0 New grouping function

Create group ：

pueue group --add youtube-download

Query the created group ：

pueue group

Output is ：

Group youtube-download (1 parallel), running: true

Add tasks to the Group ：

pueue add --immediate --group youtube-download -- youtube-dl --output "'~/Downloads/YouTube/%(title)s.%(ext)s'" --restrict-filenames "https://www.youtube.com/watch?v=Ob4Nk6gz8Ec"

Query the tasks in the Group ：

pueue status --group youtube-download

Output is ：

Group "youtube-download" (1 parallel): running
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Index   Status    Exitcode   Command                                                                                                                        Path                Start   End 
═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
 30185   Running              youtube-dl --output '~/Downloads/YouTube/%(title)s.%(ext)s' --restrict-filenames https://www.youtube.com/watch?v=Ob4Nk6gz8Ec   /Users/huzhenghui   15:49       
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 

Use the renamed follow see ：

pueue follow 30185

You can see the output , And you can see that the progress figures are changing , This has the same effect as direct execution . Some tools may display a line of swipe per number .

[youtube] Ob4Nk6gz8Ec: Downloading webpage
[download] Destination: /Users/huzhenghui/Downloads/YouTube/Calvin_Harris_Dua_Lipa_-_One_Kiss_Live_on_The_Graham_Norton_Show.f137.mp4
[download]  22.1% of 104.81MiB at 101.47KiB/s ETA 13:4445

After completion of operation , You can receive the notification set by using the callback command , The complex escape characters in the command are correctly handled in the notification content .