Originally reported of Google Code with ID 68.

<b>What steps will reproduce the problem?</b>
1. I create an isntance with drbd layout on node1 and node2, node1 is also
cluster master
2. I add a simple script (date > /tmp/date, for ex) into
/etc/ganeti/hooks/instance-failover-post.d/01-test
The instance is initially started on node1.
first gnt-instance failover (node1->node2) : everything goes well, post
hook is called on node1 and node2
second gnt-instance failover(node2 -> node1) : the hook on node2 isn't called.

Am I doing something bad ? 

Cheers,

Maxence

Originally added on 2009-08-26 09:26:15 +0000 UTC.

Originally reported of Google Code with ID 1135.

This issue is for discussion about the DRBD config leak.

Originally added on 2015-10-15 16:27:55 +0000 UTC.

Originally reported of Google Code with ID 93.

What software version are you running? Please provide the output of "gnt-
cluster --version" and "gnt-cluster version".
Software version: 2.0.6
Internode protocol: 20
Configuration format: 2000000
OS api version: 10
Export interface: 0

<b>What distribution are you using?</b>
debian testing

<b>What steps will reproduce the problem?</b>
1.create a lvm inside an encrypted disk
2.pvscan => PV /dev/block/254:0   VG vg0
3.create an instance

<b>What is the expected output? What do you see instead?</b>
ganeti will chokes saying...
Can't create block device: Can't compute PV info for vg vg0

<b>Please provide any additional information below.</b>
I was trying to migrate my cluster on an encrypted filesystem so added a
node with whole disk encrypted, fine i can also see the node and its free
disk space with "gnt-node list"! ...but if I try to create a new instance
there I get this error.

but maybe I've isolated the bug because it's very close to this report
about same issue on libvirt, please verify:
https://bugzilla.redhat.com/show_bug.cgi?id=470693

kind regards

Originally added on 2010-03-12 21:52:22 +0000 UTC.

Originally reported of Google Code with ID 1064.

What software version are you running? Please provide the output of "gnt-
cluster --version", "gnt-cluster version", and "hspace --version".

[email protected]:~# gnt-cluster --version
gnt-cluster (ganeti v2.10.7-70-g70f22ba) 2.11.6

<b>What distribution are you using?</b>

[email protected]:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 7.8 (wheezy)
Release:	7.8
Codename:	wheezy

<b>What steps will reproduce the problem?</b>
1, [email protected]:~# gluster volume create gv0 replica 2 transport tcp node1:/root/tmp1 node1:/root/tmp2
Creation of volume gv0 has been successful. Please start the volume to access data.

2, [email protected]:~# gluster volume start gv0
Starting volume gv0 has been successful

3, [email protected]:~# gnt-cluster init --vg-name=xenvg --master-netdev=eth0 --gluster-storage-dir=/var/run/ganeti/gluster -D gluster -H kvm:kernel_path=/boot/vmlinuz-3-kvmU,initrd_path=/boot/initrd-3-kvmU -N link=xen-br0 --enabled-hypervisor=kvm --enabled-disk-templates=gluster,drbd,plain cluster


4, [email protected]:~# sudo gnt-instance add -n node1 -o debootstrap+default -t gluster -s 5G instance1
Wed Apr 15 05:30:13 2015  - INFO: Resolved given name 'instance1' to 'instance1.ganeti.com'
Wed Apr 15 05:30:17 2015 * disk 0, size 5.0G
Wed Apr 15 05:30:17 2015 * creating instance disks...
Wed Apr 15 05:30:22 2015  - WARNING: Device creation failed
Failure: command execution error:
Can't create block device <Disk(type=gluster, logical_id=('loop', 'ganeti/ce24daf6-15fc-4450-abd3-f893b342b601.0'), children=None, visible as /dev/disk/0, size=5120m)> on node node1.ganeti.com for instance instance1.ganeti.com: Error while executing backend function: Path '/var/run/ganeti/gluster/ganeti/ce24daf6-15fc-4450-abd3-f893b342b601.0' is not acceptable for file storage

<b>What is the expected output? What do you see instead?</b>

Expected: Create VM on gluster backend storage successfully.
Instead: Cannot install, error happened.

<b>Please provide any additional information below.</b>

I can install VM successfully with plain disk template as follows.

[email protected]:~# sudo gnt-instance add -n node1 -o debootstrap+default -t plain -s 5G instance1
Wed Apr 15 05:37:53 2015  - INFO: Resolved given name 'instance1' to 'instance1.ganeti.com'
Wed Apr 15 05:37:57 2015 * disk 0, size 5.0G
Wed Apr 15 05:37:57 2015 * creating instance disks...
Wed Apr 15 05:37:58 2015 adding instance instance1.ganeti.com to cluster config
Wed Apr 15 05:37:59 2015  - INFO: Waiting for instance instance1.ganeti.com to sync disks
Wed Apr 15 05:37:59 2015  - INFO: Instance instance1.ganeti.com's disks are in sync
Wed Apr 15 05:38:00 2015 * running the instance OS create scripts...
Wed Apr 15 05:38:10 2015 * starting instance...
[email protected]:~# gnt-instance list
Instance             Hypervisor OS                  Primary_node     Status  Memory
instance1.ganeti.com kvm        debootstrap+default node1.ganeti.com running   128M


Cluster infos are as follows.

[email protected]:~# gnt-cluster verify
Submitted jobs 3, 4
Waiting for job 3 ...
Wed Apr 15 05:29:53 2015 * Verifying cluster config
Wed Apr 15 05:29:53 2015 * Verifying cluster certificate files
Wed Apr 15 05:29:53 2015 * Verifying hypervisor parameters
Wed Apr 15 05:29:53 2015 * Verifying all nodes belong to an existing group
Waiting for job 4 ...
Wed Apr 15 05:29:53 2015 * Verifying group 'default'
Wed Apr 15 05:29:53 2015 * Gathering data (1 nodes)
Wed Apr 15 05:29:54 2015 * Gathering disk information (1 nodes)
Wed Apr 15 05:29:54 2015 * Verifying configuration file consistency
Wed Apr 15 05:29:54 2015 * Verifying node status
Wed Apr 15 05:29:54 2015 * Verifying instance status
Wed Apr 15 05:29:54 2015 * Verifying orphan volumes
Wed Apr 15 05:29:54 2015 * Verifying N+1 Memory redundancy
Wed Apr 15 05:29:54 2015 * Other Notes
Wed Apr 15 05:29:54 2015 * Hooks Results

[email protected]:~# gnt-cluster info
Cluster name: cluster.ganeti.com
Cluster UUID: db2ddf84-451a-462e-87d7-c92c5c161d7e
Creation time: 2015-04-15 05:29:39
Modification time: 2015-04-15 05:29:46
Master node: node1.ganeti.com
Architecture (this node): 64bits (x86_64)
Tags: (none)
Default hypervisor: kvm
Enabled hypervisors: kvm
Hypervisor parameters: 
  kvm: 
    acpi: True
    boot_order: disk
    cdrom2_image_path: 
    cdrom_disk_type: 
    cdrom_image_path: 
    cpu_cores: 0
    cpu_mask: all
    cpu_sockets: 0
    cpu_threads: 0
    cpu_type: 
    disk_cache: default
    disk_type: paravirtual
    floppy_image_path: 
    initrd_path: /boot/initrd-3-kvmU
    kernel_args: ro
    kernel_path: /boot/vmlinuz-3-kvmU
    keymap: 
    kvm_extra: 
    kvm_flag: 
    kvm_path: /usr/bin/kvm
    machine_version: 
    mem_path: 
    migration_bandwidth: 32
    migration_downtime: 30
    migration_mode: live
    migration_port: 8102
    nic_type: paravirtual
    reboot_behavior: reboot
    root_path: /dev/vda1
    security_domain: 
    security_model: none
    serial_console: True
    serial_speed: 38400
    soundhw: 
    spice_bind: 
    spice_image_compression: 
    spice_ip_version: 0
    spice_jpeg_wan_compression: 
    spice_password_file: 
    spice_playback_compression: True
    spice_streaming_video: 
    spice_tls_ciphers: HIGH:-DES:-3DES:-EXPORT:-ADH
    spice_use_tls: False
    spice_use_vdagent: True
    spice_zlib_glz_wan_compression: 
    usb_devices: 
    usb_mouse: 
    use_chroot: False
    use_localtime: False
    user_shutdown: False
    vga: 
    vhost_net: False
    vnc_bind_address: 
    vnc_password_file: 
    vnc_tls: False
    vnc_x509_path: 
    vnc_x509_verify: False
    vnet_hdr: True
OS-specific hypervisor parameters: 
OS parameters: 
Hidden OSes: 
Blacklisted OSes: 
Cluster parameters: 
  candidate pool size: 10
  maximal number of jobs running simultaneously: 20
  master netdev: eth0
  master netmask: 32
  use external master IP address setup script: False
  lvm volume group: xenvg
  lvm reserved volumes: (none)
  drbd usermode helper: /bin/true
  file storage path: /srv/ganeti/file-storage
  shared file storage path: /srv/ganeti/shared-file-storage
  gluster storage path: /var/run/ganeti/gluster
  maintenance of node health: False
  uid pool: 
  default instance allocator: hail
  default instance allocator parameters: 
  primary ip version: 4
  preallocation wipe disks: False
  OS search path: /srv/ganeti/os
  ExtStorage Providers search path: /srv/ganeti/extstorage
  enabled disk templates: gluster, drbd, plain
  enabled user shutdown: False
Default node parameters: 
  exclusive_storage: False
  oob_program: 
  ovs: False
  ovs_link: 
  ovs_name: switch1
  spindle_count: 1
  ssh_port: 22
Default instance parameters: 
  default: 
    always_failover: False
    auto_balance: True
    maxmem: 128
    minmem: 128
    spindle_use: 1
    vcpus: 1
Default nic parameters: 
  default: 
    link: xen-br0
    mode: bridged
    vlan: 
Default disk parameters: 
  blockdev: 
  diskless: 
  drbd: 
    c-delay-target: 1
    c-fill-target: 0
    c-max-rate: 61440
    c-min-rate: 4096
    c-plan-ahead: 20
    data-stripes: 1
    disk-barriers: n
    disk-custom: 
    dynamic-resync: False
    meta-barriers: False
    meta-stripes: 1
    metavg: xenvg
    net-custom: 
    protocol: C
    resync-rate: 61440
  ext: 
  file: 
  gluster: 
    access: kernelspace
    host: 127.0.0.1
    port: 24007
    volume: gv0
  plain: 
    stripes: 1
  rbd: 
    access: kernelspace
    pool: rbd
  sharedfile: 
Instance policy - limits for instances: 
  bounds specs: 
    - max/0: 
        cpu-count: 8
        disk-count: 16
        disk-size: 1048576
        memory-size: 32768
        nic-count: 8
        spindle-use: 12
      min/0: 
        cpu-count: 1
        disk-count: 1
        disk-size: 1024
        memory-size: 128
        nic-count: 1
        spindle-use: 1
  std: 
    cpu-count: 1
    disk-count: 1
    disk-size: 1024
    memory-size: 128
    nic-count: 1
    spindle-use: 1
  allowed disk templates: drbd, plain, gluster
  vcpu-ratio: 4
  spindle-ratio: 32


Some logs related to gluster are in the following.
[email protected]:/var/log/ganeti# grep gluster * -R
[...]
node-daemon.log:2015-04-15 05:30:17,745: ganeti-noded pid=1054 INFO RunCmd mount -t glusterfs -o server-port=24007 127.0.0.1:/gv0 /var/run/ganeti/gluster
node-daemon.log:2015-04-15 05:30:21,946: ganeti-noded pid=1054 INFO RunCmd umount /var/run/ganeti/gluster
node-daemon.log:  File "/usr/local/share/ganeti/2.11/ganeti/storage/gluster.py", line 456, in Create
node-daemon.log:FileStoragePathError: Path '/var/run/ganeti/gluster/ganeti/ce24daf6-15fc-4450-abd3-f893b342b601.0' is not acceptable for file storage

Originally added on 2015-04-15 09:42:36 +0000 UTC.

Originally reported of Google Code with ID 1185.

 I'm running ganeti  2.15.2-1~bpo8+1 on Debian Jessie.

I'm getting the following surprising output from hbal:

# hbal -L -C 
Loaded 19 nodes, 46 instances
Group size 19 nodes, 46 instances
Selected node group: default
Initial check done: 0 bad nodes, 0 bad instances.
hbal: IntMap.!: key -1 is not an element of the map

I've discussed this with Viktor Bachraty on the mailing list, and following his instructions have created a cluster state file which can be used to test hbal during debugging using 

hbal -t cluster.state.original -C

Thanks for your help Viktor!
C.

Originally added on 2016-08-22 14:00:57 +0000 UTC.

Attachments: [cluster state file](https://storage.googleapis.com/google-code-attachments/ganeti/issue-1185/comment-0/cluster state file)

Originally reported of Google Code with ID 154.

Cluster: "gnt-cluster (ganeti v2.4.1) 2.4.1"
on 2 Debian Lenny nodes running KVM only instances.

<b>What steps will reproduce the problem?</b>
1. mount a CIFS filesystem under /srv/ganeti/export 
2. Use an OS export script like this:

#CUSTOM IMAGE EXPORT SCRIPT
set -e

. common.sh

#MCM: Executing a full raw backup of the device using dd making a file-disk-image.
if [ ! -b $blockdev ]; then
        echo "ERROR: ${blockdev} is not a Block Device."
        cleanup
        exit 1
fi
dd if=${blockdev} bs=512
EXIT_STATUS="$?"

# execute cleanups
cleanup
trap - EXIT

exit ${EXIT_STATUS}

3. run gnt-backup export -n node2 instance.name

I'm expecting the backup is done correctly but instead I get this output:

Fri Apr  8 12:50:55 2011 Shutting down instance vmbktest.
Fri Apr  8 12:51:06 2011 Creating a snapshot of disk/0 on node deborder02
Fri Apr  8 12:51:06 2011 Starting instance vmbktest
Fri Apr  8 12:51:06 2011 Exporting snapshot/0 from deborder02 to deborder02
Fri Apr  8 12:51:10 2011 snapshot/0 is now listening, starting export
Fri Apr  8 12:51:12 2011 snapshot/0 is receiving data on deborder02
Fri Apr  8 12:51:12 2011 snapshot/0 is sending data on deborder02
Fri Apr  8 12:51:17 2011 snapshot/0 sent 95M, 15.2 MiB/s
Fri Apr  8 12:52:13 2011 snapshot/0 finished receiving data
Fri Apr  8 12:52:13 2011 snapshot/0 finished sending data
Fri Apr  8 12:52:13 2011 Removing snapshot of disk/0 on node deborder02
Fri Apr  8 12:52:13 2011 Finalizing export on deborder02
Fri Apr  8 12:52:15 2011  - WARNING: Could not finalize export for instance vmbktest.or.lan on node deborder02: Error while executing backend function: [Errno 95] Operation not supported
Failure: command execution error:
Export failed, errors in export finalization

Note: into the node log i found these lines:
2011-04-08 12:52:15,317: ganeti-noded pid=4926 ERROR Error in RPC call
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/site-packages/ganeti/server/noded.py", line 152, in HandleRequest
    result = (True, method(serializer.LoadJson(req.request_body)))
  File "/usr/local/lib/python2.5/site-packages/ganeti/server/noded.py", line 428, in perspective_finalize_export
    return backend.FinalizeExport(instance, snap_disks)
  File "/usr/local/lib/python2.5/site-packages/ganeti/backend.py", line 2309, in FinalizeExport
    data=config.Dumps())
  File "/usr/local/lib/python2.5/site-packages/ganeti/utils/io.py", line 143, in WriteFile
    os.rename(new_name, file_name)
OSError: [Errno 95] Operation not supported
2011-04-08 12:52:15,319: ganeti-noded pid=4926 INFO 10.6.128.132:53061 PUT /finalize_export HTTP/1.1 200
2011-04-08 12:52:15,406: ganeti-noded pid=4927 INFO 10.6.128.132:53062 PUT /jobqueue_update HTTP/1.1 200

I tried to check if it was a python issue using this procedure:
1. I launched "python" and I typed the following commands:

import os
os.rename("/srv/ganeti/export/vmbktest.or.lan.new", "/srv/ganeti/export/vmbktest.or.lan") 

..both of them are directories and the command worked as expected.

Originally added on 2011-04-08 11:10:40 +0000 UTC.

Originally reported of Google Code with ID 617.

What software version are you running? Please provide the output of "gnt-
cluster --version", "gnt-cluster version", and "hspace --version".

Primary Node:
hspace (ganeti-htools) version v0.3.1
compiled with ghc 7.4
running on linux x86_64

gnt-cluster (ganeti v2.4.5) 2.4.5

Software version: 2.4.5
Internode protocol: 2040000
Configuration format: 2040000
OS api version: 20
Export interface: 0

Distributor ID:	Ubuntu
Description:	Ubuntu 12.04 LTS
Release:	12.04
Codename:	precise

What distribution are you using on secondary node?
Distributor ID:	Ubuntu
Description:	Ubuntu 12.10
Release:	12.10
Codename:	quantal


<b>What steps will reproduce the problem?</b>
On Primary node:
[email protected]:~# drbd-overview 
  0:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
  1:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
  2:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
  3:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
  4:??not-found??  WFConnection Primary/Unknown UpToDate/DUnknown     C r----- 
  5:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
  7:??not-found??  WFConnection Primary/Unknown UpToDate/DUnknown     C r----- 
  9:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
 10:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
 11:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
 14:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
 15:??not-found??  WFConnection Primary/Unknown UpToDate/DUnknown     C r----- 
 18:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
 20:??not-found??  WFConnection Primary/Unknown UpToDate/DUnknown     C r----- 
 22:??not-found??  WFConnection Primary/Unknown UpToDate/Inconsistent C r----- 
 23:??not-found??  WFConnection Primary/Unknown UpToDate/DUnknown     C r----- 
 34:??not-found??  WFConnection Primary/Unknown UpToDate/DUnknown     C r----- 
 38:??not-found??  WFConnection Primary/Unknown UpToDate/DUnknown     C r----- 
 39:??not-found??  WFConnection Primary/Unknown UpToDate/DUnknown     C r----- 
 41:??not-found??  WFConnection Primary/Unknown UpToDate/DUnknown     C r----- 

2.On secondary Node:
[email protected]:~# drbd-overview 
  2:??not-found??  StandAlone Secondary/Unknown Inconsistent/DUnknown r----- 
  5:??not-found??  StandAlone Secondary/Unknown Inconsistent/DUnknown r----- 
  9:??not-found??  StandAlone Secondary/Unknown Inconsistent/DUnknown r----- 
 10:??not-found??  StandAlone Secondary/Unknown Inconsistent/DUnknown r----- 
 11:??not-found??  StandAlone Secondary/Unknown Inconsistent/DUnknown r----- 
 18:??not-found??  StandAlone Secondary/Unknown Inconsistent/DUnknown r----- 
 22:??not-found??  StandAlone Secondary/Unknown Inconsistent/DUnknown r----- 

[email protected]:~# drbdsetup /dev/drbdX invalidate
/dev/drbd38: State change failed: (-2) Need access to UpToDate data


<b>What is the expected output? What do you see instead?</b>

I need to recover the nodes from split brain.

<b>Please provide any additional information below.</b>

The secondary node has been upgradeD to 12.10 quantal and the primary node is running 12.04 LTS precise. The drbd.conf is empty and there is no .res file. Any drbdadm command output " No resources defined".

The vms has been failedover to the primary node. Afraid to do anything that might bring the primary node down as this could bring the whole production down.

What is the best safe way to resolve this. 

Please help.

Originally added on 2013-11-21 08:21:21 +0000 UTC.

Originally reported of Google Code with ID 894.

What software version are you running? Please provide the output of "gnt-
cluster --version", "gnt-cluster version", and "hspace --version".

<b>What distribution are you using?</b>
# gnt-cluster --version
gnt-cluster (ganeti v2.11.3) 2.11.3
# gnt-cluster version
Software version: 2.11.3
Internode protocol: 2110000
Configuration format: 2110000
OS api version: 20
Export interface: 0
VCS version: (ganeti) version v2.11.3
# hspace --version
hspace (ganeti) version v2.11.3
compiled with ghc 7.4
running on linux x86_64
# cat /etc/debian_version
7.6
# apt-cache policy ganeti
ganeti:
  Installed: 2.11.3-2~bpo70+1
  Candidate: 2.11.3-2~bpo70+1
  Package pin: 2.11.3-2~bpo70+1
  Version table:
 *** 2.11.3-2~bpo70+1 990
        100 http://debian.xxxxxxxx.de/debian/ wheezy-backports/main amd64 Packages
        100 /var/lib/dpkg/status
     2.10.5-1~bpo70+1 990
        100 http://debian.xxxxxxxx.de/debian/ wheezy-backports/main amd64 Packages
     2.9.5-1~bpo70+1 990
        100 http://debian.xxxxxxxx.de/debian/ wheezy-backports/main amd64 Packages


<b>What steps will reproduce the problem?</b>
gnt-instance migrate -f vmX


<b>What is the expected output? What do you see instead?</b>
- Tue Jul 22 20:07:41 2014 * memory transfer progress: 600.64 %
- Tue Jul 22 20:07:53 2014 * memory transfer progress: 618.39 % 
+ "Timeout" which cancels the migration if the memory transfer progress reaches a certain limit


<b>Please provide any additional information below.</b>
The VM has got a gaming server running which seams to be modifying to much memory.

Originally added on 2014-07-22 18:15:38 +0000 UTC.

Originally reported of Google Code with ID 549.

What software version are you running? Please provide the output of "gnt-
cluster --version" and "gnt-cluster version".

[email protected]:~# gnt-cluster --version
gnt-cluster (ganeti v2.6.2) 2.6.2
[email protected]:~# gnt-cluster version
Software version: 2.6.2
Internode protocol: 2060000
Configuration format: 2060000
OS api version: 20
Export interface: 0

<b>What distribution are you using?</b>

Debian Wheezy (host), Ubuntu Precise (guest)

<b>What steps will reproduce the problem?</b>

1. Install /etc/ganeti/instance-debootstrap/hooks/grub from the examples/hooks directory. (Note: I also have the 'defaultpasswords' and 'timezone' hooks installed, and they work fine)

2. Build an Ubuntu VM. I have /etc/ganeti/instance-debootstrap/variants/precise.conf containing

MIRROR="http://archive.ubuntu.com/ubuntu/"
SUITE="precise"
EXTRA_PKGS="acpid,console-tools,udev,linux-image-virtual,openssh-server"
COMPONENTS="main,universe"
ARCH="amd64"

<b>What is the expected output? What do you see instead?</b>

The VM creation fails at the following point:

---- 8< ----
...
Searching for GRUB installation directory ... found: /boot/grub
findfs: unable to resolve 'UUID=310dafbb-8de6-42af-bf27-4c5752c99a94'
Cannot determine root device.  Assuming /dev/hda1
This error is probably caused by an invalid /etc/fstab
Searching for default file ... Generating /boot/grub/default file and setting the default boot entry to 0
Searching for GRUB installation directory ... found: /boot/grub
Testing for an existing GRUB menu.lst file ...

Could not find /boot/grub/menu.lst file. Would you like /boot/grub/menu.lst generated for you? (y/N) /usr/sbin/update-grub: line 1094: read: read error: 0: Bad file descriptor
run-parts: /etc/ganeti/instance-debootstrap/hooks/grub exited with return code 1

Current default time zone: 'Europe/London'
Local time is now:      Sun Jul 14 11:10:12 BST 2013.
Universal Time is now:  Sun Jul 14 10:10:12 UTC 2013.
---- 8< ----

As a result, the VM won't boot. You need to boot it temporarily with an external kernel, and then run grub within the VM.

<b>Please provide any additional information below.</b>

I tried changing "sda" to "vda" in the grub script, but it didn't make a difference. Also reported at https://groups.google.com/forum/#!topic/ganeti/Wmyb_5dLvaI

Originally added on 2013-08-07 18:58:05 +0000 UTC.

Originally reported of Google Code with ID 269.

What software version are you running? Please provide the output of "gnt-
cluster --version" and "gnt-cluster version".

<b>What distribution are you using?</b>
ubuntu 12.04 lts

<b>What steps will reproduce the problem?</b>

<b>1.</b>
[email protected]:~# gnt-instance add \
>         -n ubuntuxen01 \
>         -o linux+ubuntu-12.04_amd64 \
>         -t plain \
>         --disk 0:size=8G \
>         --disk 1:size=2G \
>         -B minmem=512M,maxmem=1G \
>         -H xen-pvm:kernel_path=/boot/pv-grub-4.1.2_64.gz \
>         --no-name-check \
>         --no-ip-check \
> ubuntutest01
Mon Sep 24 14:14:08 2012 * disk 0, size 8.0G
Mon Sep 24 14:14:08 2012 * disk 1, size 2.0G
Mon Sep 24 14:14:08 2012 * creating instance disks...
Mon Sep 24 14:14:09 2012 adding instance ubuntutest01 to cluster config
Mon Sep 24 14:14:09 2012  - INFO: Waiting for instance ubuntutest01 to sync disks.
Mon Sep 24 14:14:09 2012  - INFO: Instance ubuntutest01's disks are in sync.
Mon Sep 24 14:14:09 2012 * running the instance OS create scripts...
Mon Sep 24 14:15:32 2012 * starting instance...


<b>2.</b>
[email protected]:~# gnt-instance info ubuntutest01
Instance name: ubuntutest01
UUID: 8fca004e-968f-4e37-a2f9-43d972045623
Serial number: 2
Creation time: 2012-09-24 14:14:09
Modification time: 2012-09-24 14:15:32
State: configured to be up, actual state is up
  Nodes:
    - primary: ubuntuxen01.domain.intern
      group: default (UUID 32c7a1cc-5996-4f4a-aade-fa32cb67c1e8)
    - secondaries:
  Operating system: linux+ubuntu-12.04_amd64
  Allocated network port: None
  Hypervisor: xen-pvm
    - blockdev_prefix: default (sd)
    - bootloader_args: default ()
    - bootloader_path: default ()
    - cpu_mask: default (all)
    - initrd_path: default ()
    - kernel_args: default ((hd0)/boot/grub/menu.lst)
    - kernel_path: /boot/pv-grub-4.1.2_64.gz
    - reboot_behavior: default (reboot)
    - root_path: default ()
   - use_bootloader: default (False)
  Hardware:
    - always_failover: default (False)
    - auto_balance: default (True)
    - maxmem: 1024
    - memory: default (1024)
    - minmem: 512
    - spindle_use: default (1)
    - vcpus: default (2)
    - NICs:
      - nic/0: MAC: aa:00:00:09:5c:27, IP: None, mode: bridged, link: br0
  Disk template: plain
  Disks:
    - disk/0: lvm, size 8.0G
      access mode: rw
      logical_id:  system/2f801264-fe51-4e8c-9388-f5740798e97e.disk0
      on primary:  /dev/system/2f801264-fe51-4e8c-9388-f5740798e97e.disk0 (252:2)
    - disk/1: lvm, size 2.0G
      access mode: rw
      logical_id:  system/083ab68c-64dc-4689-aa16-1022f0b68778.disk1
      on primary:  /dev/system/083ab68c-64dc-4689-aa16-1022f0b68778.disk1 (252:4)

<b>3.</b>
[email protected]:~# gnt-instance grow-disk -d ubuntutest01 0 1G
2012-09-24 14:16:15,201: gnt-instance grow-disk pid=3988 cli:2238 INFO Command line: gnt-instance grow-disk -d ubuntutest01 0 1G
Mon Sep 24 14:16:15 2012 Growing disk 0 of instance 'ubuntutest01' by 1.0G to 9.0G
2012-09-24 14:16:21,232: gnt-instance grow-disk pid=3988 cli:2245 ERROR Error during command processing
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/ganeti/cli.py", line 2241, in GenericMain
    result = func(options, args)
  File "/usr/lib/pymodules/python2.7/ganeti/client/gnt_instance.py", line 673, in GrowDisk
    SubmitOrSend(op, opts)
  File "/usr/lib/pymodules/python2.7/ganeti/cli.py", line 2053, in SubmitOrSend
    return SubmitOpCode(op, cl=cl, feedback_fn=feedback_fn, opts=opts)
  File "/usr/lib/pymodules/python2.7/ganeti/cli.py", line 2030, in SubmitOpCode
    reporter=reporter)
  File "/usr/lib/pymodules/python2.7/ganeti/cli.py", line 2011, in PollJob
    return GenericPollJob(job_id, _LuxiJobPollCb(cl), reporter)
  File "/usr/lib/pymodules/python2.7/ganeti/cli.py", line 1833, in GenericPollJob
    errors.MaybeRaise(msg)
  File "/usr/lib/pymodules/python2.7/ganeti/errors.py", line 506, in MaybeRaise
    raise errcls(*args)
ProgrammerError: Can only act on disks belonging to the target instance
Unhandled Ganeti error: Can only act on disks belonging to the target instance

4.
[email protected]:~# gnt-instance info ubuntutest01
Instance name: ubuntutest01
UUID: 8fca004e-968f-4e37-a2f9-43d972045623
Serial number: 3
Creation time: 2012-09-24 14:14:09
Modification time: 2012-09-24 14:16:21
State: configured to be up, actual state is up
  Nodes:
    - primary: ubuntuxen01.domain.intern
      group: default (UUID 32c7a1cc-5996-4f4a-aade-fa32cb67c1e8)
    - secondaries:
  Operating system: linux+ubuntu-12.04_amd64
  Allocated network port: None
  Hypervisor: xen-pvm
    - blockdev_prefix: default (sd)
    - bootloader_args: default ()
    - bootloader_path: default ()
    - cpu_mask: default (all)
    - initrd_path: default ()
    - kernel_args: default ((hd0)/boot/grub/menu.lst)
    - kernel_path: /boot/pv-grub-4.1.2_64.gz
    - reboot_behavior: default (reboot)
    - root_path: default ()
    - use_bootloader: default (False)
  Hardware:
    - always_failover: default (False)
    - auto_balance: default (True)
    - maxmem: 1024
    - memory: default (1024)
    - minmem: 512
    - spindle_use: default (1)
    - vcpus: default (2)
    - NICs:
      - nic/0: MAC: aa:00:00:09:5c:27, IP: None, mode: bridged, link: br0
  Disk template: plain
  Disks:
    - disk/0: lvm, size 8.0G
      access mode: rw
      logical_id:  system/2f801264-fe51-4e8c-9388-f5740798e97e.disk0
      on primary:  /dev/system/2f801264-fe51-4e8c-9388-f5740798e97e.disk0 (252:2)
[email protected]:~#

it seams like grow-disk removes the second disk!!!

5. 
[email protected]:~# gnt-cluster verify
Submitted jobs 45, 46
Waiting for job 45 ...
Mon Sep 24 14:21:21 2012 * Verifying cluster config
Mon Sep 24 14:21:21 2012 * Verifying cluster certificate files
Mon Sep 24 14:21:21 2012 * Verifying hypervisor parameters
Mon Sep 24 14:21:21 2012 * Verifying all nodes belong to an existing group
Waiting for job 46 ...
Mon Sep 24 14:21:21 2012 * Verifying group 'default'
Mon Sep 24 14:21:21 2012 * Gathering data (1 nodes)
Mon Sep 24 14:21:23 2012 * Gathering disk information (1 nodes)
Mon Sep 24 14:21:23 2012 * Verifying configuration file consistency
Mon Sep 24 14:21:23 2012 * Verifying node status
Mon Sep 24 14:21:23 2012 * Verifying instance status
Mon Sep 24 14:21:23 2012 * Verifying orphan volumes
Mon Sep 24 14:21:23 2012   - ERROR: node ubuntuxen01.domain.intern: volume system/083ab68c-64dc-4689-aa16-1022f0b68778.disk1 is unknown
Mon Sep 24 14:21:23 2012   - ERROR: node ubuntuxen01.domain.intern: volume system/root is unknown
Mon Sep 24 14:21:23 2012   - ERROR: node ubuntuxen01.domain.intern: volume system/5de84566-cb73-4ce0-9aa6-c8d426a62d61.disk1 is unknown
Mon Sep 24 14:21:23 2012   - ERROR: node ubuntuxen01.domain.intern: volume system/swap is unknown
Mon Sep 24 14:21:23 2012 * Verifying N+1 Memory redundancy
Mon Sep 24 14:21:23 2012 * Other Notes
Mon Sep 24 14:21:23 2012   - NOTICE: 1 non-redundant instance(s) found.
Mon Sep 24 14:21:23 2012 * Hooks Results
<b>What is the expected output? What do you see instead?</b>

<b>Please provide any additional information below.</b>

Originally added on 2012-09-24 12:23:46 +0000 UTC.

Originally reported of Google Code with ID 102.

im using debian lenny trying to create a single-node cluster with LVM, no drbd.
this is the output of the command im running, and the content of my /etc/hosts. why is the 
cluster command failing?

moria:~# gnt-cluster init -b eth1 -g moria --master-netdev eth1 moria.middle-earth
Failure: prerequisites not met for this operation:
Inconsistency: this host's name resolves to 10.0.1.20,
but this ip address does not belong to this host. Aborting.
moria:~# head -n2 /etc/hosts
127.0.0.1       localhost
10.0.1.20       moria.middle-earth      moria
moria:~#

Originally added on 2010-03-31 16:54:35 +0000 UTC.

There was a previous PR for Python3.8 support, but it's not clear further Python work has taken place to support Python 3.9/3.10/3.11.

Upstream Python has both 3.8 & 3.9 series in security-only fixes.

Is there an effort to support newer Python yet, before distributions start to remove Python 3.8? (Gentoo is discussing removal of 3.8 already).

Although Windows 10 will be around until 2025, starting to think about upgrades, I saw the following error trying to install Windows 11 on Ganeti:

This PC doesn't currently meet Windows 11 system requirements Here's why: X The PC must support TPM 2.0. X The PC must support Secure Boot.

As I couldn't see any other active discussion, trying to start one here. (TPM was mentioned in #506, but didn't go anywhere.)

Xen has a live HA support, remus:

• https://wiki.xenproject.org/wiki/Remus
• https://wiki.xenproject.org/wiki/Getting_Started_with_Remus

Are there any plans for Ganeti to support this?

Hello,

We are currently upgrading our two Ganeti cluster nodes. The live migration on secondary node (node2) and server upgrade all went fine. But when doing a live migration of some instances from node1 to node2 we are experiencing problems and therefore could not proceed with the server upgrade on our first cluster node.

We have a total of 6 instances running on the cluster and the migration only fails to 3 of the instances. We compared the settings from gnt-instance info and see no big difference.

node1

• OS: Debian 11.3
• Ganeti version: 3.0.1-2
• Xen Hypervisor: 4.14.4+74-gd7b22226b5-1
• Storage: DRBD

node2

• OS: Debian 11.5
• Ganeti version: 3.0.2-1~deb11u1
• Xen Hypervisor: 4.14.5+24-g87d90d511c-1
• Storage: DRBD

Error message when migrating an instance from node1 to node2:

node2:~$ sudo gnt-instance migrate ***REMOVED***

Instance ***REMOVED*** will be migrated. Note that
migration might impact the instance if anything goes wrong (e.g. due
to bugs in the hypervisor). Continue?
y/[n]/?: y
Tue Sep 27 11:32:07 2022 Migrating instance ***REMOVED***
Tue Sep 27 11:32:08 2022 * checking disk consistency between source and target
Tue Sep 27 11:32:09 2022 * closing instance disks on node ***REMOVED-NODE2***
Tue Sep 27 11:32:09 2022 * changing into standalone mode
Tue Sep 27 11:32:10 2022 * changing disks into dual-master mode
Tue Sep 27 11:32:14 2022 * wait until resync is done
Tue Sep 27 11:32:15 2022 * opening instance disks on node ***REMOVED-NODE1*** in shared mode
Tue Sep 27 11:32:15 2022 * opening instance disks on node ***REMOVED-NODE2*** in shared mode
Tue Sep 27 11:32:16 2022 * preparing ***REMOVED-NODE2*** to accept the instance
Tue Sep 27 11:32:17 2022 * migrating instance to ***REMOVED-NODE2***
Tue Sep 27 11:32:17 2022 Migration failed, aborting
Tue Sep 27 11:32:18 2022 * closing instance disks on node ***REMOVED-NODE2***
Tue Sep 27 11:32:19 2022 * changing into standalone mode
Tue Sep 27 11:32:20 2022 * changing disks into single-master mode
Tue Sep 27 11:32:21 2022 * wait until resync is done
Failure: command execution error:
Could not migrate instance ***REMOVED***: Failed to migrate instance: Failed to migrate instance ***REMOVED***: Saving to migration stream new xl format (info 0x3/0x0/1104)
xc: info: Saving domain 5, type x86 PV
2022/09/27 11:32:17 socat[1164188] E write(7, 0x7f98d3d39000, 65536): Broken pipe
libxl: error: libxl_utils.c:510:libxl_read_exactly: file/stream truncated reading ipc msg header from domain 5 save/restore helper stdout pipe
libxl: error: libxl_exec.c:127:libxl_report_child_exitstatus: domain 5 save/restore helper [1164189] died due to fatal signal Broken pipe
migration sender: libxl_domain_suspend failed (rc=-3)
libxl: info: libxl_exec.c:117:libxl_report_child_exitstatus: migration transport process [1164188] exited with error status 1
Migration failed, resuming at sender.
xc: error: Couldn't map start_info: Internal error
libxl: error: libxl_dom_suspend.c:665:domain_resume_done: Domain 5:xc_domain_resume failed: Operation not permitted

If you need more information, please let me know. Thanks in advance.

We have cluster with a few nodes, 3 of them are marked as master-capable, the others are not. I suspect wrong behavior when changing offline flag on nodes:

node-1:~# gnt-node list -o name,master_capable,master_candidate
Node                               MasterCapable MasterC
node-1  Y             Y
node-2  Y             Y
node-3  Y             Y
node-4  N             N
node-11 N             N
node-12 N             N
node-13 N             N
node-14 N             N
node-15 N             N
node-1:~# gnt-node modify --offline=yes node-15 
Modified node node-15
 - offline -> True
node-1:~# gnt-node list -o name,master_capable,master_candidate
Node                               MasterCapable MasterC
node-1  Y             Y
node-2  Y             Y
node-3  Y             Y
node-4  N             N
node-11 N             N
node-12 N             N
node-13 N             N
node-14 N             N
node-15 N             N
node-1:~# gnt-node modify --offline=no --force node-15 
Mon Sep 12 09:25:43 2022  - INFO: Auto-promoting node to master candidate
Mon Sep 12 09:25:43 2022  - WARNING: Transitioning node from offline to online state without using re-add. Please make sure the node is healthy!
Modified node node-15
 - master_candidate -> True
 - offline -> False
node-1:~# gnt-node list -o name,master_capable,master_candidate
Node                               MasterCapable MasterC
node-1  Y             Y
node-2  Y             Y
node-3  Y             Y
node-4  N             N
node-11 N             N
node-12 N             N
node-13 N             N
node-14 N             N
node-15 N             Y
node-1:~# 

Documentation explicitly states:

Ganeti 2.4 introduces for this purpose a new master_capable flag, which (when unset) prevents nodes from being marked as master candidates, either manually or automatically.

https://docs.ganeti.org/docs/ganeti/3.0/html/admin.html#multi-site-model

We have been successfully using the move-instance Script to move instances from older clusters (e.g. based on Debian Stretch) to newer Clusters (based on Debian Bullseye / Ganeti 3.0.2). However, we can not move Instances between Debian Bullseye servers.

This happens because socat is configured to verify the TLS certificate presented by the destination node: https://github.com/ganeti/ganeti/blob/da6aba31874a87e81280c41f65541b9cb780827a/lib/impexpd/init.py#L91

However, with recent socat versions verification also includes matching the hostname to the certificate CN/SAN entries. For the connection, the destination node's ip address is used, but the cluster certificate always contains ganeti.example.com. This is hardcoded in the constants: https://github.com/ganeti/ganeti/blob/da6aba31874a87e81280c41f65541b9cb780827a/src/Ganeti/Constants.hs#L605

I see multiple solutions to this problem:

• we can supply each node with a valid certificate (which contains the hostname and all primary/secondary IP addresses) and use that for the import socket server
• we can make the verify switch configurable and leave it up to the user (with verify=1 being always broken)
• set verify=0 to at least allow people to migrate instances again
• something else

What would you suggest?