Schedule migrate export

[Sebastieng]

Hi,

As i didn't find anything using google, i made a script that could be use in order to schedule an upgrade export (each week for example), and backup files to an ftp server :

Those variables must be change; adjust according to your Checkpoint and FTP configuration :

• HOST
• USER
• PASSWD

• Create the script export.sh into /usr/local/bin folder :

Paste this content into export.sh (don't forget to adapt:

#!/bin/bash -f
source /etc/profile.d/CP.sh

#VARIABLES
DATE=`date +%Y%m%d`
FILE=$HOSTNAME"_"$DATE"_export"
FOLDER=$FWDIR/bin/upgrade_tools/

echo ******Process Migrate Export `date +'%d.%m.%y %H:%M:%S'` STARTED******


echo Migrate export $FILE started
#UPGRADE_EXPORT
echo File will be $FOLDER$FILE.tgz
$FWDIR/bin/upgrade_tools/migrate export -n $FOLDER/$FILE
echo Migrate export $FILE finished

#Waiting for migrate export to finished
echo  waiting 2  minutes
sleep 2m
echo waiting finished


echo FTP $FILE  started
#FTP
cd $FOLDER
HOST=HOST
USER=USER
PASSWD=PASSWD
ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
bin
put $FILE.tgz
quit
END_SCRIPT

echo FTP $FILE finished

#REMOVE
echo Remove of  $FOLDER/$FILE.tgz started

rm $FWDIR/bin/upgrade_tools/$FILE.tgz

echo Remove of  $FOLDER/$FILE.tgz finished

echo ******Process Migrate export `date +'%d.%m.%y %H:%M:%S'` FINISHED******

exit

#Script written by Sebastien GUILMER

• Make the script executable :

chmod +x export.sh

• Modify the crontab :

crontab –e

• Paste the line (don't forget to adapt) :

* * * * 0 /usr/local/bin/export.sh >> /usr/local/bin/CRONexport.txt

The script running without error give that :

[Expert@Mgmt:0]# ./export.sh
******Process Migrate Export 09.05.14 14:22:26 STARTED******
Migrate export Mgmt_20140509_export started
File will be /opt/CPsuite-R77/fw1/bin/upgrade_tools/Mgmt_20140509_export.tgz
Migrate export Mgmt_20140509_export finished
waiting 2 minutes...
waiting finished
FTP Mgmt_20140509_export started
FTP Mgmt_20140509_export finished
Remove of /opt/CPsuite-R77/fw1/bin/upgrade_tools//Mgmt_20140509_export.tgz started
Remove of /opt/CPsuite-R77/fw1/bin/upgrade_tools//Mgmt_20140509_export.tgz finished
******Process Migrate export 09.05.14 14:23:28 FINISHED******
[Expert@Mgmt:0]#

Try this in lab before deploying in production !

Best regards,

Sébastien G

[jflemingeds]

just a quick FYI. You can source

/etc/profile.d/CP.sh

Then you don't need to change the script everytime you upgrade. I would also put the script in a private dir outside of the checkpoint dirs so that the script location wouldn't change after and upgrade as well.

nice script!

[Sebastieng]

just a quick FYI. You can source

/etc/profile.d/CP.sh

Then you don't need to change the script everytime you upgrade. I would also put the script in a private dir outside of the checkpoint dirs so that the script location wouldn't change after and upgrade as well.

nice script!

Hi jflemingeds,

Thank for the information, it's very usefull !

So i can put the script in the home directory of admin, right ?

[jflemingeds]

I'm not a fan of scripts in the home dir. I think /usr/local/bin would be a good idea, but really its up to you.

[Sebastieng]

Thank you,

I have edited my post with the "best practice" provided :-), Thank you !

[aweldon]

The script looks great, I went to test it after plugging in the appropriate information and I'm getting a /bin/bash error for invalid options although it shows -f as being a shell option.

[jflemingeds]

The script looks great, I went to test it after plugging in the appropriate information and I'm getting a /bin/bash error for invalid options although it shows -f as being a shell option.

What is it your expecting -f to do? I think think you need any argument on the bash line for your script to work.

[isharted]

i would recommend using scp with key authentication so you don't need to store a password in plain text. other than that, this is what my script looks like

[aweldon]

What is it your expecting -f to do? I think think you need any argument on the bash line for your script to work.

Honestly I was just trying to test the script. I pulled the -f out and it errors with:
bash: /usr/local/bin/export.sh: /bin/bash^M: bad interpreter: No such file or directory

[jflemingeds]

Honestly I was just trying to test the script. I pulled the -f out and it errors with:
bash: /usr/local/bin/export.sh: /bin/bash^M: bad interpreter: No such file or directory

I think you've somehow got DOS/Windows systle new line ver unix in export.sh


The way you get the ^M is by doing

CTRL-V
CTRL-M

sed 's/^M//g' export.sh > new.export.sh

Then try to run new.export.sh

If that doesn't work let us know.

[aweldon]

Thanks, that command did not work so I ran dos2unix export.sh and all seems well now.

(I used notepad++ and saved as a Unix script without editing the EOL > Unix/OSX format)

[jflemingeds]

Thanks, that command did not work so I ran dos2unix export.sh and all seems well now.

(I used notepad++ and saved as a Unix script without editing the EOL > Unix/OSX format)

good deal!

[DaveCullen86]

Thanks a lot, just used this myself! Took me a little while to fine tune to my requirements as my scripting and linux skills are not all that but I now have a daily upgrade_export going to my FTP server. Just got to work out a way to make it delete any files a week old now :)

Thanks again - Dave

[The_Dude]

Is there any way to build some error checking into this? What would happen if a step failed or didn't ftp successful and continues on script and deleted the files?

[dsb.nepo]

Is there any way to build some error checking into this? What would happen if a step failed or didn't ftp successful and continues on script and deleted the files?

This scrip contains some error checking
https://www.cpug.org/forums/showthre...remote-trigger
It is an older script and needs some adjustments e.g.
- replace /opt/CPshrd-R65/tmp/.CPprofile.sh with the path from your version
- replace "echo | ${EXPT} ${FILE_TGZ}" with "echo y | ${EXPT} ${FILE_TGZ}"

Since it uses scp it is best to generate an ssh key for the transfer user and place the public part on the target system in the $targetuser/.ssh/authorized_keys file and test if ssh login works without password

[slowfood27]

Here 's another one. It works both for SmartCenter and Multi-Domain. It uses SCP as well to copy files. It check the available disk space before starting the backup process and notifies by mail.
Feel free to adopt to your needs.

#!/bin/bash
#
# Filename: /home/admin/scripts/backup_export.sh
# ================================================== =============================
#
# Shell script to provide regular backup by meamns of exporting the
# Check Point database of a SmartCenter or Multi Domain Server.
# The backup type can be specified as either daily, weekly, monthly or quarterly
# According to the backup type, the number of backup files kept by the purging
# process varies.
# If the disk space is not sufficient, no backup will be done
# It will send an email to an ADMIN group, if the backup has failed
#
# Copyright WellComm AG, 2014, Version 1.0
# -------------------------------------------------------------------------------
# get correct environment first
#
. /etc/bashrc
#
# -------------------------------------------------------------------------------
# Set some variables then
#
BACKUPTYPE=""
FILESTOKEEP=0
DOMAIN_TYPE=0
DOMTYPE="sc"
DOMAIN_NAME="SmartCenter"
MDSWASTE="CPSmart*.tgz"
MAILSERVER="192.168.10.1"
MAILADDRESS="securiy-group@customer.com"
FACILITY="%BCK-EXP-intro"
BACKUPTYPES=([0]=daily [1]=weekly [2]=monthly [3]=quarterly)
FILEVERSIONS=([0]=5 [1]=4 [2]=3 [3]=4)
#--------------------------------------------------------------------------------
#
# Make sure that input parameter is not blank
#
if [ "$1" = "" ] ; then
echo "$FACILITY Input parameter must not be blank, please specify either daily, weekly, monthly or quarterly"
exit
fi
#--------------------------------------------------------------------------------
# Check which type of backup is selected and assign files to purge
#
for idx in 0 1 2 3
do
if [ ${BACKUPTYPES[$idx]} = "$1" ] ; then
BACKUPTYPE=${BACKUPTYPES[$idx]}
echo "$FACILITY We will do $BACKUPTYPE backup"
fi
if [ ${BACKUPTYPES[$idx]} = "$BACKUPTYPE" ] ; then
FILESTOKEEP=${FILEVERSIONS[$idx]}
# echo "$FACILITY We will keep $FILESTOKEEP file(s)"
fi
done
# -------------------------------------------------------------------------------
# Make sure the variables are correct, otherwise quit
#
if [ "$BACKUPTYPE" = "" ] ; then
echo "$FACILITY Illegal input parameter $1 for backup operation specified, exiting"
exit
fi
if [ $FILESTOKEEP = 0 ] ; then
echo "$FACILITY Illegal parameter $FILESTOKEEP for files to keep in purging operation, exiting"
exit
fi
#
FACILITY="%BCK-EXP"_"$BACKUPTYPE"
# -------------------------------------------------------------------------------
# Check if we are a SmartCenter- or a Multi-Domain-Server and set the
# start/stop commands accordingly
#
if [ "$(command -v mdsstat)" = "" ] ; then
echo "$FACILITY Identified $(hostname) as a SmartCenter"
CMDPRFX="cp"
else
echo "$FACILITY Identified $(hostname) as a Multi Domain Server"
CMDPRFX="mds"
DOMTYPE=$CMDPRFX
DOMAIN_TYPE=1
DOMAIN_NAME="Multi Domain"
fi
STOPCMD=$CMDPRFX"stop"
STARTCMD=$CMDPRFX"start"
# -------------------------------------------------------------------------------
#
BACKUPDIR=/var/export/
#
BACKUPFILE=${BACKUPDIR}Exported_$(hostname)_${DOMT YPE}_${BACKUPTYPE}_$(date +%Y-%m-%d).tgz
#
# set alert level 80% of disk space is default
ALERT=80
#
# -------------------------------------------------------------------------------
#
function check_disk_space() {
while read output;
do
#echo $output
usep=$(echo $output | awk '{ print $2}' | cut -d'%' -f1)
partition=$(echo $output | awk '{print $3}')
spaceleft=$(echo $output | awk '{print $1}')
runstat=0
if [ $usep -ge $ALERT ] ; then
echo "$FACILITY Running out of space on partition $partition ($usep% used), only $spaceleft left on server $(hostname), $(date)"
# mail -s "Alert: Almost out of disk space $usep%" $ADMIN
runstat=1
fi
done
return $runstat
}
#
# ================================================== =============================
#
# start our main task
#
#
echo "$FACILITY Starting $BACKUPTYPE Check Point export backup on server $(hostname), $(date)"
df -PH $BACKUPDIR | grep -vE "^Filesystem" | awk '{print $4 " " $5 " " $6}' | check_disk_space
rc=$?
if [ $rc = 0 ] ; then
echo "$FACILITY Partition size for $BACKUPDIR is o.k., continuing"
else
echo "$FACILITY Partition size for $BACKUPDIR is too small, exiting"
exit
fi
# -------------------------------------------------------------------------------
#
# stop the Management Server and perform the backup
#
echo "$FACILITY Stopping the $DOMAIN_NAME Server"
#
$STOPCMD
#
echo "$FACILITY Exporting database files to $BACKUPFILE"
if [ $DOMAIN_TYPE = 0 ] ; then
#
echo "$FACILITY we will now perform: $FWDIR/bin/upgrade_tools/migrate export -n $BACKUPFILE "
$FWDIR/bin/upgrade_tools/migrate export -n $BACKUPFILE
else
MDS_FILE=$(date +%d%b%Y-)*.mdsbk.tgz
echo "$FACILITY we will now perform: mds_backup -b -l -d /var/export >> /dev/null"
mds_backup -b -l -d /var/export >> /dev/null
fi
#echo "$FACILITY Backup file name is $BACKUPFILE"
#echo "$FACILITY Wrote $BACKUPTYPE Check Point export backup on server $(hostname), $(date)" > $BACKUPFILE
rc=$?
# -------------------------------------------------------------------------------
# Purge the directory if the backup operation was successful
#
if [ $rc = 0 ] ; then
if [ $DOMAIN_TYPE = 1 ] ; then
echo "$FACILITY We will now rename $BACKUPDIR$MDS_FILE to $BACKUPFILE"
mv $BACKUPDIR$MDS_FILE $BACKUPFILE
FILE=$BACKUPDIR$MDSWASTE
stat -t $FILE >/dev/null 2>&1
rc=$?
if [ $rc = 0 ] ; then
echo "$FACILITY Cleaning up MDS Backup files $FILE"
rm $FILE
fi
fi
cd $BACKUPDIR
echo "$FACILITY Purging files in backup directory $BACKUPDIR"
files_in_dir=`ls *$BACKUPTYPE* | wc -l`
files_to_delete=`expr $files_in_dir - $FILESTOKEEP`
if [ $files_to_delete -gt 0 ]; then
ls -r *$BACKUPTYPE* | tail -n $files_to_delete | xargs rm
rc=$?
if [ $rc -ne 0 ]; then
echo "$FACILITY An error ocurred deleting the files in $BACKUPDIR"
else
echo "$FACILITY $files_to_delete file(s) deleted in $BACKUPDIR."
fi
else
echo "$FACILITY no files to delete while purging the directory $BACKUPDIR!"
rc=0
fi
if [ $rc = 0 ] ; then
echo "$FACILITY Sucessfully completed $BACKUPTYPE Check Point export backup on server $(hostname), $(date)"
else
echo "$FACILITY Failed to purge $BACKUPTYPE Check Point export backups on server $(hostname), $(date)"
fi
else
echo "$FACILITY Failed to write $BACKUPTYPE Check Point export backup on server $(hostname), $(date)"
echo "$FACILITY Failed to write $BACKUPTYPE Check Point export backup on server $(hostname), $(date)" > $HOME/scripts/backup_state.txt
echo "Return status is $rc"
$FWDIR/bin/sendmail -t $MAILSERVER -s 'Check Point Backup Failure' -f sendmail@$HOSTNAME $MAILADDRESS < $HOME/scripts/backup_state.txt
fi
#
# -------------------------------------------------------------------------------
# restart the mgmt server
#
echo "$FACILITY Re-starting the $DOMAIN_NAME Server"
$STARTCMD
#
# well, that's it, we are done
#
# -------------------------------------------------------------------------------

[slowfood27]

Here 's another one. It works both for SmartCenter and Multi-Domain. It check the available disk space before starting the backup process and notifies by mail.
Feel free to adopt to your needs. If you need scp copy with keys as a script, drop me a PM

#!/bin/bash
#
# Filename: /home/admin/scripts/backup_export.sh
# ================================================== =============================
#
# Shell script to provide regular backup by meamns of exporting the
# Check Point database of a SmartCenter or Multi Domain Server.
# The backup type can be specified as either daily, weekly, monthly or quarterly
# According to the backup type, the number of backup files kept by the purging
# process varies.
# If the disk space is not sufficient, no backup will be done
# It will send an email to an ADMIN group, if the backup has failed
#
# Copyright WellComm AG, 2014, Version 1.0
# -------------------------------------------------------------------------------
# get correct environment first
#
. /etc/bashrc
#
# -------------------------------------------------------------------------------
# Set some variables then
#
BACKUPTYPE=""
FILESTOKEEP=0
DOMAIN_TYPE=0
DOMTYPE="sc"
DOMAIN_NAME="SmartCenter"
MDSWASTE="CPSmart*.tgz"
MAILSERVER="192.168.10.1"
MAILADDRESS="securiy-group@customer.com"
FACILITY="%BCK-EXP-intro"
BACKUPTYPES=([0]=daily [1]=weekly [2]=monthly [3]=quarterly)
FILEVERSIONS=([0]=5 [1]=4 [2]=3 [3]=4)
#--------------------------------------------------------------------------------
#
# Make sure that input parameter is not blank
#
if [ "$1" = "" ] ; then
echo "$FACILITY Input parameter must not be blank, please specify either daily, weekly, monthly or quarterly"
exit
fi
#--------------------------------------------------------------------------------
# Check which type of backup is selected and assign files to purge
#
for idx in 0 1 2 3
do
if [ ${BACKUPTYPES[$idx]} = "$1" ] ; then
BACKUPTYPE=${BACKUPTYPES[$idx]}
echo "$FACILITY We will do $BACKUPTYPE backup"
fi
if [ ${BACKUPTYPES[$idx]} = "$BACKUPTYPE" ] ; then
FILESTOKEEP=${FILEVERSIONS[$idx]}
# echo "$FACILITY We will keep $FILESTOKEEP file(s)"
fi
done
# -------------------------------------------------------------------------------
# Make sure the variables are correct, otherwise quit
#
if [ "$BACKUPTYPE" = "" ] ; then
echo "$FACILITY Illegal input parameter $1 for backup operation specified, exiting"
exit
fi
if [ $FILESTOKEEP = 0 ] ; then
echo "$FACILITY Illegal parameter $FILESTOKEEP for files to keep in purging operation, exiting"
exit
fi
#
FACILITY="%BCK-EXP"_"$BACKUPTYPE"
# -------------------------------------------------------------------------------
# Check if we are a SmartCenter- or a Multi-Domain-Server and set the
# start/stop commands accordingly
#
if [ "$(command -v mdsstat)" = "" ] ; then
echo "$FACILITY Identified $(hostname) as a SmartCenter"
CMDPRFX="cp"
else
echo "$FACILITY Identified $(hostname) as a Multi Domain Server"
CMDPRFX="mds"
DOMTYPE=$CMDPRFX
DOMAIN_TYPE=1
DOMAIN_NAME="Multi Domain"
fi
STOPCMD=$CMDPRFX"stop"
STARTCMD=$CMDPRFX"start"
# -------------------------------------------------------------------------------
#
BACKUPDIR=/var/export/
#
BACKUPFILE=${BACKUPDIR}Exported_$(hostname)_${DOMT YPE}_${BACKUPTYPE}_$(date +%Y-%m-%d).tgz
#
# set alert level 80% of disk space is default
ALERT=80
#
# -------------------------------------------------------------------------------
#
function check_disk_space() {
while read output;
do
#echo $output
usep=$(echo $output | awk '{ print $2}' | cut -d'%' -f1)
partition=$(echo $output | awk '{print $3}')
spaceleft=$(echo $output | awk '{print $1}')
runstat=0
if [ $usep -ge $ALERT ] ; then
echo "$FACILITY Running out of space on partition $partition ($usep% used), only $spaceleft left on server $(hostname), $(date)"
# mail -s "Alert: Almost out of disk space $usep%" $ADMIN
runstat=1
fi
done
return $runstat
}
#
# ================================================== =============================
#
# start our main task
#
#
echo "$FACILITY Starting $BACKUPTYPE Check Point export backup on server $(hostname), $(date)"
df -PH $BACKUPDIR | grep -vE "^Filesystem" | awk '{print $4 " " $5 " " $6}' | check_disk_space
rc=$?
if [ $rc = 0 ] ; then
echo "$FACILITY Partition size for $BACKUPDIR is o.k., continuing"
else
echo "$FACILITY Partition size for $BACKUPDIR is too small, exiting"
exit
fi
# -------------------------------------------------------------------------------
#
# stop the Management Server and perform the backup
#
echo "$FACILITY Stopping the $DOMAIN_NAME Server"
#
$STOPCMD
#
echo "$FACILITY Exporting database files to $BACKUPFILE"
if [ $DOMAIN_TYPE = 0 ] ; then
#
echo "$FACILITY we will now perform: $FWDIR/bin/upgrade_tools/migrate export -n $BACKUPFILE "
$FWDIR/bin/upgrade_tools/migrate export -n $BACKUPFILE
else
MDS_FILE=$(date +%d%b%Y-)*.mdsbk.tgz
echo "$FACILITY we will now perform: mds_backup -b -l -d /var/export >> /dev/null"
mds_backup -b -l -d /var/export >> /dev/null
fi
#echo "$FACILITY Backup file name is $BACKUPFILE"
#echo "$FACILITY Wrote $BACKUPTYPE Check Point export backup on server $(hostname), $(date)" > $BACKUPFILE
rc=$?
# -------------------------------------------------------------------------------
# Purge the directory if the backup operation was successful
#
if [ $rc = 0 ] ; then
if [ $DOMAIN_TYPE = 1 ] ; then
echo "$FACILITY We will now rename $BACKUPDIR$MDS_FILE to $BACKUPFILE"
mv $BACKUPDIR$MDS_FILE $BACKUPFILE
FILE=$BACKUPDIR$MDSWASTE
stat -t $FILE >/dev/null 2>&1
rc=$?
if [ $rc = 0 ] ; then
echo "$FACILITY Cleaning up MDS Backup files $FILE"
rm $FILE
fi
fi
cd $BACKUPDIR
echo "$FACILITY Purging files in backup directory $BACKUPDIR"
files_in_dir=`ls *$BACKUPTYPE* | wc -l`
files_to_delete=`expr $files_in_dir - $FILESTOKEEP`
if [ $files_to_delete -gt 0 ]; then
ls -r *$BACKUPTYPE* | tail -n $files_to_delete | xargs rm
rc=$?
if [ $rc -ne 0 ]; then
echo "$FACILITY An error ocurred deleting the files in $BACKUPDIR"
else
echo "$FACILITY $files_to_delete file(s) deleted in $BACKUPDIR."
fi
else
echo "$FACILITY no files to delete while purging the directory $BACKUPDIR!"
rc=0
fi
if [ $rc = 0 ] ; then
echo "$FACILITY Sucessfully completed $BACKUPTYPE Check Point export backup on server $(hostname), $(date)"
else
echo "$FACILITY Failed to purge $BACKUPTYPE Check Point export backups on server $(hostname), $(date)"
fi
else
echo "$FACILITY Failed to write $BACKUPTYPE Check Point export backup on server $(hostname), $(date)"
echo "$FACILITY Failed to write $BACKUPTYPE Check Point export backup on server $(hostname), $(date)" > $HOME/scripts/backup_state.txt
echo "Return status is $rc"
$FWDIR/bin/sendmail -t $MAILSERVER -s 'Check Point Backup Failure' -f sendmail@$HOSTNAME $MAILADDRESS < $HOME/scripts/backup_state.txt
fi
#
# -------------------------------------------------------------------------------
# restart the mgmt server
#
echo "$FACILITY Re-starting the $DOMAIN_NAME Server"
$STARTCMD
#
# well, that's it, we are done
#
# -------------------------------------------------------------------------------