API target for development

[Bob_Zimmerman]

Most of my development work so far has been against a 2200 which I personally own. It has a perpetual license, but it's sometimes a little unpredictable. The API service sometimes crashes. It has a lot of unique state from all the changes I've made on it while building stuff. This makes it okay for human testing, but really not great for automated testing (it's hard to write a unit test if the API target can move).

Clearly the 15-day plug-n-play eval is well-suited for this. Just spin up a new VM with a known config, and run the automated tests against that, right? Well, it turns out the 15-day eval starts counting down when you first boot GAiA. If you do stuff like installing patches or adding your SSH keys, you're out of luck. The image is only good for 15 days. And the installation is kind of hard to automate. You have to type stuff like the password you want for admin, and the initial IP address.

Blink is the solution to this, right? Well ... not really. It lets you automate some of the installation, but you still end up with a different web UI certificate and SSH host keys every time you reimage the system. Automated tests against a new certificate every time? Pretty gross. Key change prompt every time you connect via SSH? Even worse.



I ended up playing a little trick on GAiA. I ran the R80.40 installer, then at the end when it asks to reboot, I instead powered off the VM. Next, I booted it with a Linux LiveCD (I happened to use Devuan. Anything which understands LVM and XFS should be fine.) and mounted the LVM volumes. This let me make changes to the filesystem before the first boot. Specifically, I made these changes:

1. Copied /web/conf/server.key and /web/conf/server.crt from a working VM to this new one
2. Copied /etc/ssh/ssh_host* from a working VM to this new one
3. Made /home/admin/.ssh/authorized_keys and added my workstation's public key to it

Note that /home/admin/.ssh must be mode 700, and /home/admin/.ssh/authorized_keys must be mode 600.

With the files in place, I shut down the VM, removed the LiveCD from its optical drive, and took another snapshot. Booted the VM, and checked the eval expiration date via 'show installer status'. Using the hypervisor's CLI tool, I powered the VM off, restored it to the snapshot, and powered it on again. Checked the eval expiration date again, and it was different! Success!

Note that when you restore to the snapshot, it is a completely unconfigured GAiA system. If you give it your SSH key via the LiveCD as I described above, you can easily copy over a config_system file and use that to run the initial wizard. From there, you can easily also copy over the latest CPUSE, jumbo, and so on, and install them. I am further using some mgmt_cli commands to insert some initial data for development purposes.



Kind of a headache, but it's the best way I've found so far to get disposable SmartCenters or MDSs without a huge amount of manual interaction.

[Bob_Zimmerman]

I eventually decided using snapshots for this is too slow. I have a ludicrously powerful desktop (2x Xeon X5675 [3.06 GHz, 6 cores plus hyperthreading], 96 GB of RAM), and it was still taking over 20 minutes to run config_system and another 20 to install the jumbo. I flattened the snapshot and cloned the pre-first-boot VM. Now I have a VM named "R80.40" which is a base R80.40 installation with the modifications I made above. It has never been booted. I then have my build script clone this VM into a new working VM. config_system is down to 7:30 m:ss, and installing the jumbo is down to a hair over ten minutes. I'm currently using a SATA SSD, which is clearly the next bottleneck (waiting for a good deal on an Intel P4618, but if somebody wanted to donate an Optane card, I wouldn't say no!). Still, cutting build time by ~45% is nothing to sneeze at.

Here's my basic build script for the moment:

#!/bin/zsh
sourceImage="R80.40"
systemIPAddress="10.0.3.240"
localBuildDir=~/OneDrive/CPMDS_Build
systemName="TestMDS"

success() {
echo " [\e[0;92m\e[1mDone at $(date "+%H:%M:%S")\e[0m]"
}

sshCmd() {
ssh admin@"${systemIPAddress}" "$argv" >>/tmp/"${systemName}".output 2>>/tmp/"${systemName}".output
}

scpFile() {
for file in "$argv[@]";do
scp "$file" admin@"${systemIPAddress}": >>/tmp/"${systemName}".output 2>>/tmp/"${systemName}".output
done
}

jumboName="$(cd "${localBuildDir}";ls *.tgz | grep JUMBO)"

echo -n "" > /tmp/"${systemName}".output
echo "Starting at $(date "+%H:%M:%S")"

echo -n "Resetting VM to the cleanly installed state ..."
vboxmanage controlvm "${systemName}" poweroff >>/tmp/"${systemName}".output 2>>/tmp/"${systemName}".output
vboxmanage unregistervm "${systemName}" --delete >>/tmp/"${systemName}".output 2>>/tmp/"${systemName}".output
vboxmanage clonevm "${sourceImage}" --mode=machine --name="${systemName}" --register >>/tmp/"${systemName}".output 2>>/tmp/"${systemName}".output
false;while [ $? -ne 0 ];do
vboxmanage startvm "${systemName}" >>/tmp/"${systemName}".output 2>>/tmp/"${systemName}".output
done
success

echo -n "Waiting for the system to finish booting ..."
( exit 255 );while [ $? -gt 1 ];do
sleep 1
sshCmd lock database override
done
sshCmd set user admin shell /bin/bash
sshCmd clish -c \"save config\"
success

echo -n "Running first-time config wizard. This takes a while ..."
scpFile "${localBuildDir}"/"${systemName}".config_system.txt
sshCmd config_system -f "${systemName}".config_system.txt
systemIPAddress="$(grep ipaddr_v4 "${localBuildDir}"/"${systemName}".config_system.txt | cut -d'=' -f2)"
sshCmd rm "${systemName}".config_system.txt
success

echo -n "Copying CPUSE and jumbo to the system ..."
scpFile "${localBuildDir}"/*.tgz
success

echo -n "Updating CPUSE ..."
sshCmd clish -c \"lock database override\"
sshCmd clish -c \"installer agent install $(cd "${localBuildDir}";ls *.tgz | grep DeploymentAgent)\"
sshCmd rm DeploymentAgent_\*
success

echo -n "Sleeping five minutes to give services a chance to quiesce ..."
sleep 300
success

echo -n "Installing jumbo ..."
sshCmd clish -c \"lock database override\"
sshCmd clish -c \"installer import local $jumboName\"
echo "y" | sshCmd clish -c \"installer install $jumboName\"
sshCmd rm $jumboName
sshCmd 'while true;do sleep 1;done'
success

echo -n "Running API script ..."
false;while [ $? -ne 0 ];do
sleep 60
sshCmd mgmt_cli -r true show hosts limit 1
done
sshCmd mgmt_cli -r true set api-settings accepted-api-calls-from \"all ip addresses that can be used for gui clients\" automatic-start true
sshCmd api restart
sshCmd mgmt_cli -r true add domain name FirstCMA servers.0.name CMA1 servers.0.ipv4-address 10.0.3.241 servers.0.multi-domain-server "${systemName}" servers.0.active true servers.0.type \"management server\"
success

The important variables are at the top. "sourceImage" needs to match the name of the template VM in VirtualBox. systemIPAddress needs to be the IP you gave it in the GAiA installation wizard. localBuildDir is the directory which contains your config_system file, deployment agent, and jumbo. Finally, systemName is the name you want the new VM to get. This name is also used to find the appropriate config_system file in the localBuildDir. In my specific example, the config_system file is named "TestMDS.config_system.txt".

Towards the middle (right after config_system runs), you can see it searches the config_system file for the ipaddr_v4 field and sets systemIPAddress to that value. VMs cloned from a template will always have the original address you specified, but this allows multiple VMs to be built from that single template (e.g., if you want to try management HA, or MDSM/MDSC separation). If you need to move the VM to a different network, that's the time to do it. Note that all VMs produced from a given template will have the same SSH keys.

All output from the remote system is logged to /tmp/<systemName>.output. If something doesn't work, or takes longer than expected, check there.