automated MDS backup

[Wiktor]

Hello team
hopefully this is in correct section of forum
I need to do specific task in job, which is do monthly backup and send to storage server.

Because im not a linux a guy was thinking about script which would look like this and add to cron:

mdsstop && cd /local/backups/ && mds_backup -g -l && mdsstart

Later probably I'll add tar command to compress files into single one and then send via scp to storage server then delete it.


Does it make sense to do it like this? The point is that it needs to be automated fully, so there are no questions during execution of scrit like "press y to logout users". etc.

[Bob_Zimmerman]

I normally use mds_backup -b -i -l. The b sets batch mode, which doesn't prompt for anything. The i includes the rule hit counts. The l (lowercase L) excludes logs (I have separate MLMs, so this is just a belt-and-suspenders to keep the size tolerable).

I've been working on a script to fully automate this, complete with email notification. I don't have SCP automated yet, but the email, creation of the backup, and the removal of old backups past the retention period all works:

========================================
THIS SCRIPT IS OBSOLETE.
Leaving it here for educational purposes.
A new version has been posted below.
========================================

#!/bin/env bash
MTA="<mail.example.com>"
RECIPIENTS="<someAddress@example.com; secondAddress@example.com>"
SUBJECT_DATE="$(date --iso-8601)"
WORKING_DIR="/var/log/mds_backup"
NUMBER_TO_KEEP=1

. /etc/profile.d/CP.sh
############################################################
## Send a notice the backup is starting, then wait five
## minutes to give people a chance to see it.
printf "From: root@$(hostname)
To: ${RECIPIENTS}
Subject: MDS backup for ${SUBJECT_DATE}
mds_backup beginning on $(hostname) in five minutes, at $(date -d '+5 minutes' --iso-8601=seconds)." \
| /sbin/sendmail --host="${MTA}" --read-envelope-from -t
sleep 300s

############################################################
## Run the backup.
if [ ! -d ${WORKING_DIR} ];then mkdir ${WORKING_DIR};fi
touch markerFile
MDS_ERROR=$( { mds_backup -b -i -l -d ${WORKING_DIR}>/dev/null; } 2>&1 )
MDS_EXIT=$?
cd ${WORKING_DIR}
for fileName in $(ls *.mdsbk.tgz | grep -v $(hostname)); do
	dateStamp=$(echo $fileName | cut -d. -f1)
	mv ${fileName} ${dateStamp}.$(hostname).mdsbk.tgz
	done
cd "$OLDPWD"

############################################################
## SCP code here.
NEW_BACKUP="$(find ${WORKING_DIR} -newer markerFile -name '*.mdsbk.tgz')"
## SCP_EXIT=$?
SCP_EXIT=0

############################################################
## If the SCP worked, clean up old files to leave only the
## number specified in FILES_TO_KEEP. Older files are
## removed first.
FILES_REMOVED=""
if [ ${SCP_EXIT} -eq 0 ];then
	NUMBER_TO_REMOVE=$(($(ls -t ${WORKING_DIR}/*.mdsbk.tgz | wc -l)-${NUMBER_TO_KEEP}))
	if [ ${NUMBER_TO_REMOVE} -gt 0 ];then
		FILES_REMOVED=$(ls -t ${WORKING_DIR}/*.mdsbk.tgz | tail -n ${NUMBER_TO_REMOVE})
		echo "${FILES_REMOVED}" | xargs -L 1 /bin/rm
		fi
	fi
/bin/rm markerFile

############################################################
## Report backup status to the admins.
printf "From: root@$(hostname)
To: ${RECIPIENTS}
Subject: MDS backup for ${SUBJECT_DATE}
mds_backup finished on $(hostname) at $(date --iso-8601=seconds).

mds_backup exit code: ${MDS_EXIT}
SCP exit code: ${SCP_EXIT}

mds_backup STDERR:
############################################################
${MDS_ERROR}
############################################################

Files removed:
${FILES_REMOVED}" \
| /sbin/sendmail --host="${MTA}" --read-envelope-from -t

========================================
THIS SCRIPT IS OBSOLETE.
Leaving it here for educational purposes.
A new version has been posted below.
========================================

To use this, you need to replace <mail.example.com> and <someAddress@example.com; secondAddress@example.com> with values for your environment. You can put the script anywhere, and run it with cron.

The removal happens after the SCP, so you can set NUMBER_TO_KEEP to 0 if you don't want to retain any local copies. I like to retain at least one local in case I need to import without rebuilding the whole MDS.

It puts the files in /var/log/mds_backup by default. If the directory doesn't exist, the script will create it.

It renames the files to include the hostname of the MDS. My organization's fileservers are managed by a separate team, so I include this to help them find me if they have questions about the enormous files.

[Bob_Zimmerman]

I just updated my MDS past the versions in sk163300, which changed mds_backup to no longer gzip the final tar file. That broke my file renaming logic. Testing a fix.

Edited to add: This should work for versions which emit .mdsbk.tar files instead of .mdsbk.tgz. Also fixed a stupid ordering issue with the markerFile. The marker was created after running the mds backup:

#!/bin/env bash
MTA="<mail.example.com>"
RECIPIENTS="<someAddress@example.com; secondAddress@example.com>"
WORKING_DIR="/var/log/mds_backup"
NUMBER_TO_KEEP=1

. /etc/profile.d/CP.sh
############################################################
## Send a notice the backup is starting, then wait five
## minutes to give people a chance to see it.
printf "From: root@$(hostname)
To: ${RECIPIENTS}
Subject: MDS backup for $(date --iso-8601)
mds_backup beginning on $(hostname) in five minutes, at $(date -d '+5 minutes' --iso-8601=seconds)." \
| /sbin/sendmail --host="${MTA}" --read-envelope-from -t
sleep 300s

############################################################
## Run the backup.
if [ ! -d ${WORKING_DIR} ];then mkdir ${WORKING_DIR};fi
touch ${WORKING_DIR}/markerFile
MDS_ERROR=$( { mds_backup -b -i -l -d ${WORKING_DIR}>/dev/null; } 2>&1 )
MDS_EXIT=$?
cd ${WORKING_DIR}
for fileName in $(ls *.mdsbk.tgz | grep -v $(hostname)); do
	dateStamp=$(echo $fileName | cut -d. -f1)
	mv ${fileName} ${dateStamp}.$(hostname).mdsbk.tgz
	done
for fileName in $(ls *.mdsbk.tar | grep -v $(hostname)); do
	dateStamp=$(echo $fileName | cut -d. -f1)
	mv ${fileName} ${dateStamp}.$(hostname).mdsbk.tar
	done
cd "$OLDPWD"

############################################################
## Check the API afterwards. Sometimes it seems to die
## during an mds_backup. If dead, restart it.
api status>/dev/null
API_EXIT=$?
if [ ${API_EXIT} -ne 0 ];then api start>/dev/null;fi

############################################################
## SCP code here.
NEW_BACKUP="$(find ${WORKING_DIR} -newer markerFile -name '*.mdsbk.*')"
## SCP_EXIT=$?
SCP_EXIT=0

############################################################
## If the SCP worked, clean up old files to leave only the
## number specified in FILES_TO_KEEP. Older files are
## removed first.
FILES_REMOVED=""
if [ ${SCP_EXIT} -eq 0 ];then
	NUMBER_TO_REMOVE=$(($(ls -t ${WORKING_DIR}/*.mdsbk.* | wc -l)-${NUMBER_TO_KEEP}))
	if [ ${NUMBER_TO_REMOVE} -gt 0 ];then
		FILES_REMOVED=$(ls -t ${WORKING_DIR}/*.mdsbk.* | tail -n ${NUMBER_TO_REMOVE})
		echo "${FILES_REMOVED}" | xargs -L 1 /bin/rm
		fi
	fi
/bin/rm markerFile

############################################################
## Report backup status to the admins.
printf "From: root@$(hostname)
To: ${RECIPIENTS}
Subject: MDS backup for $(date --iso-8601)
mds_backup finished on $(hostname) at $(date --iso-8601=seconds).

mds_backup exit code: ${MDS_EXIT}
After backup, API was $(if [ ${API_EXIT} -ne 0 ];then printf "not running. Restarted.";else printf "running.";fi)
SCP exit code: ${SCP_EXIT}

mds_backup STDERR:
############################################################
${MDS_ERROR}
############################################################

Files removed:
${FILES_REMOVED}" \
| /sbin/sendmail --host="${MTA}" --read-envelope-from -t

[EricAnderson]

Nice work, Zimmie (as always).

-E

[Bob_Zimmerman]

Thanks for the comment! I'm never sure if anybody else cares about this kind of thing.

[Wiktor]

This is amazing what you did Bob. Big thanks on it.

I did so far something basic script before i found out someone responded to this thread. I've added this to be executed by cron.
cat script.sh
source /etc/profile.d/CP.sh
touch /home/cronuser/$(date +%y-%m-%d)mds_backup.log
date >> /home/cronuser/$(date +%y-%m-%d)mds_backup.log
echo test OK >> /home/cronuser/$(date +%y-%m-%d)mds_backup.log
mds_backup -l -b -d /home/admin/MDSbackup >> /home/cronuser/$(date +%y-%m-%d)mds_backup.log


The point is that overall the mds_backup was performed, but it didnt log anything. do you know why? Script even didnt created file (touch /home/cronuser/$(date +%y-%m-%d)mds_backup.log).
Is it because im using CP.sh as source? Or did i mess something?

[Wiktor]

The point is that overall the mds_backup was performed, but it didnt log anything. do you know why? Script even didnt created file (touch /home/cronuser/$(date +%y-%m-%d)mds_backup.log).
Is it because im using CP.sh as source? Or did i mess something?

Edit:
I've checked today files on MDS and i see that log was created, but according to my worker when he did ls /home/cronuser | grep *.log file was not there during backup process. I somehow doubt that file was not there during backup and was created after finishing backup.
I clearly see in log, that echo which is executed in my script right before backup was done properly, so im not sure if its neither my worker is lying or maybe indeed file was not visible during backup process

cat 20-07-12backup.log | less
Sun Jul 12 20:00:02 BST 2020
test OK

Multi-Domain Server Backup
=======================================
<output omitted>

[Bob_Zimmerman]

The file should be created as soon as you touch it, and it should have contents as soon as the >> is run. My bet would be time zone confusion (maybe he checked before the script had run?) or node confusion (maybe he checked on the secondary MDS?).

That said, the way your script is written, it runs '$(date +%y-%m-%d)' several separate times. It would probably be a better idea to run that once and store the result in a variable, which you then use each of the other times. As it is, if your script runs at 23:59 of some day, it could end up writing its output to two different files.

[Wiktor]

Thats what I though so, but still he was really confident about it. I'll verify backup this week by myself.

Anyways Bob, you are totally right about the date part in my script. I just never did any scripting on real work this is early draft which I'll fix soon. Thanks on this.

Finally i have last question.
I want to do SCP automated authentciaton with ssh key.
I suppose that i need to run ssh-keygen -b 4096 command on MDS and then copy key to remote server right?
Is there any chance that generating pub key on MDS may break anything? Like connectivity between MDS and other domains? I suppose it will not, but want to be 100% sure

[Bob_Zimmerman]

SSH keys are a user-level thing. Check Point doesn't use them directly for anything, and they won't interfere with anything Check Point does.

I'm working on SCP stuff myself (specifically, still building the file server to push the backups to), but it's really easy in general. You just specify the identity file you want to use (probably /home/admin/.ssh/id_rsa), and specify what you want to copy to where. The exit code should be 0 for success or non-0 for failure. My code already tests for success before removing any files, but I currently hard-code the variable to 0. Just remove 'SCP_EXIT=0', uncomment '## SCP_EXIT=$?', and add your SCP code immediately above that line. 'NEW_BACKUP' will contain the path to the file which was just created.

[Wiktor]

Hi Bob

few things
1) I have watched cronbackup this week and i know why my coworker didnt see backup log. grep *.log seems to not work as i thougt, however file was created from the beggining and was visible when doing ls command
2) when mds_backup was locking databases i've seen there was one failure

Failed to connect to group04, server is down
CPmds-R77/customers/Group04/
CPmds-R77/customers/Group04/CPSmartLog-R77/
CPmds-R77/customers/Group04/CPSmartLog-R77/tmp/
CPmds-R77/customers/Group04/opsec_pull_cert
CPmds-R77/customers/Group04/CPSmartLog-R77/smartlog_server
CPmds-R77/customers/Group04/CPSmartLog-R77/conf
<ommited rest of output>

Does it means that Group 04 was backuped but MDS didnt manage to lockdatabase?
What can be the reason that MDS coudnt connect to database?
3) Im trying to configure sending mails with your script, but im getting followed error

./sendmail.sh
sendmail: cannot locate host <my_company.com>: Name or service not known
sendmail: could not send mail

Doest it means im using wrong MTA or what?

Sorry for all these questions. I never used linux services at this level

[Wiktor]

3) Im trying to configure sending mails with your script, but im getting followed error

Doest it means im using wrong MTA or what?

Sorry for all these questions. I never used linux services at this level

Please ignore. I didnt realise i had to remove "<>" from script to make it work.

So if possible please answer to my second question and i guess this is all

[Bob_Zimmerman]

Ah. Yeah. By convention, brackets indicate optional arguments in UNIX/Linux, and less-than and greater-than indicate mandatory arguments. In both cases, the enclosing characters need to be removed as well.

As for the "Failed to connect to group04", that's a good question. I don't know the answer, unfortunately. I supported Provider-1 in the TAC back in 2010 or so, but didn't touch it again for about eight years after I left. Just now getting back into it.

[cciesec2006]

Ah. Yeah. By convention, brackets indicate optional arguments in UNIX/Linux, and less-than and greater-than indicate mandatory arguments. In both cases, the enclosing characters need to be removed as well.

As for the "Failed to connect to group04", that's a good question. I don't know the answer, unfortunately. I supported Provider-1 in the TAC back in 2010 or so, but didn't touch it again for about eight years after I left. Just now getting back into it.

Can't believe that I've not worked on Checkpoint for over two years. Now I am just working on PaloAlto. Anyway, here is my script I use to backup my MDS R77.30 every two days. I use cron to automate the backup process:

[Expert@P1:0]# more backup_mds
#!/bin/sh -x
. /etc/profile.d/CP.sh

DATE="`/bin/date +%b_%d_%Y_%Hh_%Mm`"
FILENAME="mds_backup_`uname -n`_$DATE"
LOG=/var/tmp/backup_log_$DATE
BACKUPDIR=/var/backup/tmp
LOCALSTORAGE=/var/backup/storage


### First, do no harm by entering the /var/tmp directory
cd /var/tmp

### Set the mds environment with the mdsenv command

mdsenv

### create /var/backup/tmp if one does not exist


if ! [ -d /var/backup ]
then
echo "Backup folder missing, creating" >> $LOG
mkdir /var/backup
fi
if ! [ -d /var/backup/tmp ]
then
echo "Backup tmp folder missing, creating" >> $LOG
mkdir /var/backup/tmp
fi
if ! [ -d /var/backup/storage ]
then
echo "Backup storage folder missing, creating" >> $LOG
mkdir /var/backup/storage
fi
if ! [ -d /var/backup/log ]
then
echo "Backup log folder missing, creating" >> $LOG
mkdir /var/backup/log
fi


### Enter $BACKUPDIR directory
cd $BACKUPDIR

### Remove EVERYTHING inside $BACKUPDIR directory
echo -e "Cleaning /var/backup/tmp" >> $LOG
rm -rf /var/backup/tmp/*

### Create today mds_backups with today time directory
echo -e "Creating $BACKUPDIR/$FILENAME directory\n" >> $LOG
mkdir $BACKUPDIR/$FILENAME

### Enter $BACKUPDIR/$FILENAME

cd $BACKUPDIR/$FILENAME


### Gather important system information

/bin/clish -c 'show configuration' >> $BACKUPDIR/$FILENAME/configuration.txt

# STOP MDS
echo -e "==========\nStopping MDS\n==========\n" >> $LOG
$MDSDIR/scripts/mdsstat >> $LOG
$MDSDIR/scripts/mdsstop >> $LOG
sleep 5

# Pass 2
$MDSDIR/scripts/mdsstat >> $LOG
$MDSDIR/scripts/mdsstop >> $LOG
sleep 5


## Perform backup
echo -e "\n==================\nBeginning mdsbackup\n==================\n" >> $LOG
#echo y | $MDSDIR/scripts/mds_backup -b -d $BACKUPDIR/$FILENAME 2>> $LOG >> /dev/null
echo y | $MDSDIR/scripts/mds_backup -s -L all -b -d $BACKUPDIR/$FILENAME 2>> $LOG >> /dev/null
echo -e "\n==================\nCompleted mdsbackup\n==================\n" >> $LOG

### Pack up
echo -e "Packing up Backup Folder" >> $LOG
md5sum $BACKUPDIR/$FILENAME/* > $BACKUPDIR/$FILENAME/md5sum.txt
tar -cvf $BACKUPDIR/$FILENAME.tar $BACKUPDIR/$FILENAME
mv $BACKUPDIR/$FILENAME.tar $LOCALSTORAGE/

echo -e "\nStarting MDS\n" >> $LOG
#$MDSDIR/scripts/mdsstart >> $LOG
sleep 60
#sleep 30
$MDSDIR/scripts/mdsstat >> $LOG

### This is where I send my MDS backup to an external Linux Storage
echo -e "Off-loading backup files to network" >> $LOG
scp -i /etc/scripts/.ssh/id_rsa $LOCALSTORAGE/$FILENAME.tar root@4.2.2.2:/data/fwmgr >> $LOG



### Removing Compress, fixed in 75.47
################################
#### Compress Rule Hit Table ###
################################

#echo -e "\n================\nCompressing Rule Hit Table\n================\n" >> $LOG

mdsenv 172.31.167.10
mcd conf
rm $FWDIR/conf/hit_count_rules_table.sqlite.backup

cp $FWDIR/conf/hit_count_rules_table.sqlite $FWDIR/conf/hit_count_rules_table.sqlite.backup
ls -l $FWDIR/conf/hit_count_rules_table.sql* >> $LOG

$FWDIR/conf/hit_count_unification_tool.sh >> $LOG
ls -l $FWDIR/conf/hit_count_rules_table.sql* >> $LOG
sleep 5
$FWDIR/conf/hit_count_unification_tool.sh >> $LOG
ls -l $FWDIR/conf/hit_count_rules_table.sql* >> $LOG
sleep 5
$FWDIR/conf/hit_count_unification_tool.sh >> $LOG
ls -l $FWDIR/conf/hit_count_rules_table.sql* >> $LOG
sleep 5
#$FWDIR/conf/hit_count_unification_tool.sh >> $LOG
#ls -l $FWDIR/conf/hit_count_rules_table.sql* >> $LOG
#sleep 5
#$FWDIR/conf/hit_count_unification_tool.sh >> $LOG
#ls -l $FWDIR/conf/hit_count_rules_table.sql* >> $LOG

echo -e "\n================\nCompleted Compressing Rule Hit Table\n================\n" >> $LOG

mv $LOG /var/backup/log

#echo "REBOOT Phase" >> $LOG
#/sbin/shutdown -r now

[dbrown3611]

cciesec2006:
Nice to see you posting. I've been out of CP for three years now, gave up the working world and joined the leisure class.

Curious to know about your PA experiences and the good/bad when compared to CP. Hope that life is treating you well and offering a heartfelt thanks for all your contributions to this forum. Know that I learned a hell of a lot from you.

Best regards,
Dave

[Wiktor]

For me personally PA has a lot better support from TAC rather than CP. No problems with SIC, separate MGMT plane from user traffic plane, some QoL upgrades. For example, you dont need to create rules to allow IPSEC ports on external interface. Just assign IPSEC config to interface, and voila.

Stuff i dont like is that Panorama (mgmt server for multipel firewalls) is running only in browser. No dedicated client, meaning sometimes it gets sloopy. Also if you have big envrioment, reviewing logs sometimes is slow. You cant open 2 windows with separate search logs. Rather than that you need to login twice to panorama.

NEvertheless, i think PA outstanded CP in firewall market.