\
Go Back   RightScale Forums > RightScale

Reply
 
Thread Tools Display Modes
  #1  
Old 04-18-2012, 12:03 PM
robert.piskule robert.piskule is offline
Junior Member
Cumulus
 
Join Date: Jul 2011
Posts: 14
Unhappy EBS Continuous backups - 11H1 [rev 5] fails

Does anyone have any ideas?

I have a volume at /dev/sdj, mounted to /persistant. It is formatted as xfs.


INPUTS:
Code:
EBS_BACKUP_FREQUENCY Information
text:5 * * * *

EBS_BACKUP_KEEP_DAILY Information
text:14

EBS_BACKUP_KEEP_LAST Information
text:60

EBS_BACKUP_KEEP_MONTHLY Information
text:12

EBS_BACKUP_KEEP_WEEKLY Information
text:6

EBS_BACKUP_KEEP_YEARLY Information
text:2

EBS_LINEAGE Information
text:backup-liveserver

EBS_LINEAGE_PREFIX_OVERRIDE Information
No value/Ignore

EBS_MOUNT_POINT Information
text:/persistant

EBS_STRIPE_COUNT Information
text:1
 
EBS_TERMINATE_SAFETY Information
text:Override the dropdown and set to "off" to really run this script
 
EBS_TOTAL_VOLUME_GROUP_SIZE Information
text:1



CRON LOG:
Code:
syncing filesystem..                                                            
                                                                                
Filesystem synched.                                                             
running xfs_freeze -f /persistant                                               
                                                                                
Performing RightScale API call to create a new snapshot                         
CAUGHT EXCEPTION in: create_ebs_backup({:max_snaps=>60, :keep_weeklies=>6, :pre\
fix_override=>"backup-liveserver", :keep_monthlies=>12, :devices=>"", :keep_yea\
rlies=>2, :lineage=>"backup-liveserver", :suffix=>"201204181444", :description=\
>"Snapshot created by RightScale DB tools on instance i-a00c65c7.", :keep_daili\
es=>14})                                                                        
#<RestClient::RequestFailed: HTTP status code 422>                              
#<Net::HTTPClientError:0x2aaaac58cfb0>                                          
{"aws_ids":[],"error":"No devices specified\n"}                                 
RESPONSE BODY: {"aws_ids":[],"error":"No devices specified\n"}                  
running xfs_freeze -u /persistant                                               
                                                                                
running xfs_freeze -f /persistant                                               
                                                                                
Performing RightScale API call to create a new snapshot                         
CAUGHT EXCEPTION in: create_ebs_backup({:max_snaps=>60, :keep_weeklies=>6, :pre\
fix_override=>"backup-liveserver", :keep_monthlies=>12, :devices=>"", :keep_yea\
rlies=>2, :lineage=>"backup-liveserver", :suffix=>"201204181444", :description=\
>"Snapshot created by RightScale DB tools on instance i-a00c65c7.", :keep_daili\
es=>14})                                                                        
#<RestClient::RequestFailed: HTTP status code 422>                              
#<Net::HTTPClientError:0x2aaaac57c020>                                          
{"aws_ids":[],"error":"No devices specified\n"}                                 
RESPONSE BODY: {"aws_ids":[],"error":"No devices specified\n"}                  
running xfs_freeze -u /persistant                                               
                                                                                
/opt/rightscale/ebs/backupEBS_stripe.rb:116: uninitialized constant RightScale:\
:RemoteExecException (NameError)                                                
=========  Starting New Backup =========                                        
Wed Apr 18 14:44:09 -0400 2012                                                  
-D 14 -W 6 -M 12 -Y 2                                                           
/opt/rightscale/ebs/backupEBS_stripe.rb -s '201204181444' -l  'backup-liveserve\
r' -n 'backup-liveserver'  -c 'backup-liveserver' -t 'rs_snapshot' -p /persista\
nt -m 60 -D 14 -W 6 -M 12 -Y 2                                                  
Creating snapshot                                                               
WARNING: snapshot failed - retrying                                             
Creating snapshot                                                               
WARNING: snapshot failed - retrying                                             
Backup execution failed
Reply With Quote
  #2  
Old 04-18-2012, 12:29 PM
Ryan - RightScale Ryan - RightScale is offline
RightScale Representative
Cumulonimbus
 
Join Date: Mar 2009
Posts: 197
Send a message via Skype™ to Ryan - RightScale
Default

You need to use the device instead of the mount point for the snapshot. When we make the API call for the EBS snapshot that's the parameter that we'll pass, not the mount point.

Notice ":devices=>""" is what is currently being passed when the API call is being made.
Reply With Quote
  #3  
Old 04-18-2012, 12:58 PM
robert.piskule robert.piskule is offline
Junior Member
Cumulus
 
Join Date: Jul 2011
Posts: 14
Default

So, do I change:
Code:
EBS_MOUNT_POINT Information
text:/persistant
To:
Code:
EBS_MOUNT_POINT Information
text:/dev/sdj
?
Reply With Quote
  #4  
Old 04-18-2012, 01:26 PM
Ryan - RightScale Ryan - RightScale is offline
RightScale Representative
Cumulonimbus
 
Join Date: Mar 2009
Posts: 197
Send a message via Skype™ to Ryan - RightScale
Default

Actually, I think you have the input configured correctly. We parse the device path from the mount point or default to it being our standard LVM logical volume. It looks like somewhere the EBS tools aren't able to find the physical device path by checking the fstab entry for your mount point. Do you currently have an entry for the volume in fstab? If not it'll fail to find the device. It might make more sense for us to switch from fstab to mtab in the future, however this is how it is currently written in the 11H1 EBS Stripe Toolbox.
Reply With Quote
  #5  
Old 04-18-2012, 01:55 PM
robert.piskule robert.piskule is offline
Junior Member
Cumulus
 
Join Date: Jul 2011
Posts: 14
Default

I'll look into it-- I probably.

Out of curiosity, I assume rightscale does not automatically do this when a volume is attached? (which means volumes will mess up on reboot, not terminate and start)?
Reply With Quote
  #6  
Old 04-18-2012, 02:48 PM
robert.piskule robert.piskule is offline
Junior Member
Cumulus
 
Join Date: Jul 2011
Posts: 14
Default

I updated my scripts to automatically insert the fstab entry. Here's what I have. Still no go:

Code:
syncing filesystem..

Filesystem synched.
running xfs_freeze -f /persistant

Creating snapshot
Performing RightScale API call to create a new snapshot
CAUGHT EXCEPTION in: create_ebs_backup({:max_snaps=>60, :keep_weeklies=>6, :prefix_override=>"backup-liveserver", :keep_monthlies=>12, :devices=>"", :keep_yearlies=>2, :lineage=>"backup-liveserver", :suffix=>"201204181405", :description=>"Snapshot created by RightScale DB tools on instance .", :keep_dailies=>14})
#<RestClient::RequestFailed: HTTP status code 422>
#<Net::HTTPClientError:0x2aaaac5896f8>
{"aws_ids":[],"error":"No devices specified\n"}
RESPONSE BODY: {"aws_ids":[],"error":"No devices specified\n"}
WARNING: snapshot failed - retrying
running xfs_freeze -u /persistant
And here is my fstab file. Note that there is a new line before /dev/sdj. I didn't think that would be a problem.

Code:
/dev/sda1  /         ext3    defaults        0 0                                
/dev/sdb   /mnt      ext3    defaults        0 0                                
none       /dev/pts  devpts  gid=5,mode=620  0 0                                
none       /dev/shm  tmpfs   defaults        0 0                                
none       /proc     proc    defaults        0 0                                
none       /sys      sysfs   defaults        0 0                                
                                                                            
/dev/sdj /persistant xfs defaults 0 0
Reply With Quote
  #7  
Old 04-23-2012, 09:06 AM
robert.piskule robert.piskule is offline
Junior Member
Cumulus
 
Join Date: Jul 2011
Posts: 14
Default

Anyone have any ideas?
Reply With Quote
  #8  
Old 04-23-2012, 02:31 PM
Ryan - RightScale Ryan - RightScale is offline
RightScale Representative
Cumulonimbus
 
Join Date: Mar 2009
Posts: 197
Send a message via Skype™ to Ryan - RightScale
Default

It looks like I was wrong again. If you look in the tools it looks like the stripe tools search for the devices by scanning for the LVM Physical Devices.

From the tools you can see both methods:

Code:
# uses mount_point to grab device out of /etc/fstab
    def get_device_from_fstab(mount_point = @mount_point, fstab_path = "/etc/fstab")
      fstab = IO.readlines(fstab_path)
      fstab.each do |entry|
        dev, mp, _, _ = entry.split(/[\t\s]/)
        return dev if mp == mount_point
      end
      nil
    end

    def get_physical_device_names
# use volume-group name to detect 'physical' device names
      pvs = `pvdisplay -c`
      pvs.collect do |d|
        pd, vg, _ = d.split(/:/)
        pd.gsub!(/^\s+/,"")
        pd if vg == self.vgname
      end.reject { |s| s.nil? }
    end
This means that to use the stripe tools you'll need the volumes to be valid physical volumes in LVM.

I haven't tried this, but you might be able to hack around this by changing line 101 of the backupEBS_stripe.rb file from

Code:
create_opts[:devices] = ebs.disk.get_physical_device_names.join(",")
to

Code:
create_opts[:devices] = ebs.get_device_from_fstab
Reply With Quote
  #9  
Old 04-23-2012, 02:48 PM
Ryan - RightScale Ryan - RightScale is offline
RightScale Representative
Cumulonimbus
 
Join Date: Mar 2009
Posts: 197
Send a message via Skype™ to Ryan - RightScale
Default

Although, the hack would only theoretically work if you have a single volume that is formatted xfs, not an actual stripe set.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 04:22 AM.