Nimrod's Tech Blog » Gatling – Scaling Out Your Load Tests

Scaling out your load tests using Gatling, Packer, and a cloud provider (in this example we will use DigitalOcean and Amazon EC2.

*This post assumes that you know what Gatling is all about and need to scale your existing scenario to run from several machines / from different networks.

*want to try this out? use this referral link and get free 10$ in digital ocean (you can fire up many machines with 10$ )

Why scale out ?
-Problem I’ve encountered with Gatling is the max amount of connection 1 machine can send to a server in order to stress test it.
-In a fast explanation the default setting for a ubuntu machine is around 470~ connections (link)
-with some tweaking on the machine (setting virtual network interfaces and increasing the port range) you can get more (link)
-firing up a new VM on the cloud and running reuqests from it is easy, fast and cheap!

First lets create a image for our “Gatling Node”, we can deploy this image multiple times and the images are set to enable ssh from 1 to another, this image template is a packer json – link to packer json file

Image contains

ubuntu14.04 as base image
all necessary pre-executes to run Gatling scenario
SSH keys to enable SSH from nodes to eachother
Gatling installation

{
  "builders": [
    {
      "type": "digitalocean",
      "api_token": "XXXX",
      "image": "ubuntu-14-04-x64",
      "region" : "nyc3",
      "size" : "1gb",
      "droplet_name" : "gatlingNode"
    },
    {
      "type": "amazon-ebs",
      "access_key": "XXX",
      "secret_key": "XXX",
      "region": "us-east-1",
      "source_ami": "ami-b8067ed0",
      "instance_type": "t2.micro",
      "ssh_username": "ubuntu",
      "ami_name": "gatlingNode"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "sudo apt-get update",
        "sudo apt-get install -y unzip",
        "sudo apt-get install -y default-jre",
        "wget https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/2.1.4/gatling-charts-highcharts-bundle-2.1.4-bundle.zip -P /tmp",
        "mkdir ~/gatling",
        "unzip /tmp/gatling-charts-highcharts-bundle-2.1.4-bundle.zip -d ~/gatling/",
        "ssh-keygen -f ~/.ssh/id_rsa -t rsa -N ''",
        "touch ~/.ssh/authorized_keys",
        "cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys",
        "touch ~/.ssh/config",
        "echo \"Host *\" >> ~/.ssh/config",
        "echo \"  StrictHostKeyChecking no\" >> ~/.ssh/config",
        "sync",
        "sleep 120",
        "echo \"finished running the script\""
      ]
    }
  ]
}

{

"builders": [

{

"type": "digitalocean",

"api_token": "XXXX",

"image": "ubuntu-14-04-x64",

"region" : "nyc3",

"size" : "1gb",

"droplet_name" : "gatlingNode"

{

"type": "amazon-ebs",

"access_key": "XXX",

"secret_key": "XXX",

"region": "us-east-1",

"source_ami": "ami-b8067ed0",

"instance_type": "t2.micro",

"ssh_username": "ubuntu",

"ami_name": "gatlingNode"

}

"provisioners": [

{

"type": "shell",

"inline": [

"sudo apt-get update",

"sudo apt-get install -y unzip",

"sudo apt-get install -y default-jre",

"wget https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/2.1.4/gatling-charts-highcharts-bundle-2.1.4-bundle.zip -P /tmp",

"mkdir ~/gatling",

"unzip /tmp/gatling-charts-highcharts-bundle-2.1.4-bundle.zip -d ~/gatling/",

"ssh-keygen -f ~/.ssh/id_rsa -t rsa -N ''",

"touch ~/.ssh/authorized_keys",

"cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys",

"touch ~/.ssh/config",

"echo \"Host *\" >> ~/.ssh/config",

"echo \" StrictHostKeyChecking no\" >> ~/.ssh/config",

"sync",

"sleep 120",

"echo \"finished running the script\""

]

}

]

}

After your new cluster is ready you can use this script to run your scenario on all the nodes link to script

Main steps of the script

Copy simulation to all remote hosts
Running simulations all remote hosts
Running simulation on localhost
Collect all simulation to localhost
Generate report with aggregated data
Open browser with the report

the script is pretty strait forward and well documented on each step :

#!/bin/bash
##################################################################################################################
#Gatling scale out/cluster run script:
#If you are using the packer images this should run out of the box
#Before running this script some assumptions are made:
#1) Public keys were exchange inorder to ssh with no password promot (ssh-copy-id on all remotes)
#2) Check  read/write permissions on all folders declared in this script.
#3) Gatling installation (GATLING_HOME variable) is the same on all hosts
#4) Assuming all hosts has the same user name (if not change in script)
##################################################################################################################

#Assuming same user name for all hosts
USER_NAME='nimrod'

#Remote hosts list
HOSTS=( 192.168.28.24 192.123.123.12 180.123.98.1)

#Assuming all Gatling installation in same path (with write permissions)
GATLING_HOME=/gatling/gatling-charts-highcharts-1.5.6
GATLING_SIMULATIONS_DIR=$GATLING_HOME/user-files/simulations
GATLING_RUNNER=$GATLING_HOME/bin/gatling.sh

#Change to your simulation class name
SIMULATION_NAME='nimrodstech.GatlingClusterTest'

#No need to change this
GATLING_REPORT_DIR=$GATLING_HOME/results/
GATHER_REPORTS_DIR=/gatling/reports/

echo "Starting Gatling cluster run for simulation: $SIMULATION_NAME"

echo "Cleaning previous runs from localhost"
rm -rf $GATHER_REPORTS_DIR
mkdir $GATHER_REPORTS_DIR
rm -rf $GATLING_REPORT_DIR

for HOST in "${HOSTS[@]}"
do
  echo "Cleaning previous runs from host: $HOST"
  ssh -n -f $USER_NAME@$HOST "sh -c 'rm -rf $GATLING_REPORT_DIR'"
done

for HOST in "${HOSTS[@]}"
do
  echo "Copying simulations to host: $HOST"
  scp -r $GATLING_SIMULATIONS_DIR/* $USER_NAME@$HOST:$GATLING_SIMULATIONS_DIR
done

for HOST in "${HOSTS[@]}"
do
  echo "Running simulation on host: $HOST"
  ssh -n -f $USER_NAME@$HOST "sh -c 'nohup $GATLING_RUNNER -nr -s $SIMULATION_NAME > /gatling/run.log 2>&1 &'"
done

echo "Running simulation on localhost"
$GATLING_RUNNER -nr -s $SIMULATION_NAME

echo "Gathering result file from localhost"
ls -t $GATLING_REPORT_DIR | head -n 1 | xargs -I {} mv ${GATLING_REPORT_DIR}{} ${GATLING_REPORT_DIR}report
cp ${GATLING_REPORT_DIR}report/simulation.log $GATHER_REPORTS_DIR


for HOST in "${HOSTS[@]}"
do
  echo "Gathering result file from host: $HOST"
  ssh -n -f $USER_NAME@$HOST "sh -c 'ls -t $GATLING_REPORT_DIR | head -n 1 | xargs -I {} mv ${GATLING_REPORT_DIR}{} ${GATLING_REPORT_DIR}report'"
  scp $USER_NAME@$HOST:${GATLING_REPORT_DIR}report/simulation.log ${GATHER_REPORTS_DIR}simulation-$HOST.log
done

mv $GATHER_REPORTS_DIR $GATLING_REPORT_DIR
echo "Aggregating simulations"
$GATLING_RUNNER -ro reports

#using macOSX
open ${GATLING_REPORT_DIR}reports/index.html

#using ubuntu
#google-chrome ${GATLING_REPORT_DIR}reports/index.html

#!/bin/bash

##################################################################################################################

#Gatling scale out/cluster run script:

#If you are using the packer images this should run out of the box

#Before running this script some assumptions are made:

#1) Public keys were exchange inorder to ssh with no password promot (ssh-copy-id on all remotes)

#2) Check read/write permissions on all folders declared in this script.

#3) Gatling installation (GATLING_HOME variable) is the same on all hosts

#4) Assuming all hosts has the same user name (if not change in script)

##################################################################################################################

#Assuming same user name for all hosts

USER_NAME='nimrod'

#Remote hosts list

HOSTS=( 192.168.28.24 192.123.123.12 180.123.98.1)

#Assuming all Gatling installation in same path (with write permissions)

GATLING_HOME=/gatling/gatling-charts-highcharts-1.5.6

GATLING_SIMULATIONS_DIR=$GATLING_HOME/user-files/simulations

GATLING_RUNNER=$GATLING_HOME/bin/gatling.sh

#Change to your simulation class name

SIMULATION_NAME='nimrodstech.GatlingClusterTest'

#No need to change this

GATLING_REPORT_DIR=$GATLING_HOME/results/

GATHER_REPORTS_DIR=/gatling/reports/

echo "Starting Gatling cluster run for simulation: $SIMULATION_NAME"

echo "Cleaning previous runs from localhost"

rm -rf $GATHER_REPORTS_DIR

mkdir $GATHER_REPORTS_DIR

rm -rf $GATLING_REPORT_DIR

for HOST in "${HOSTS[@]}"

echo "Cleaning previous runs from host: $HOST"

ssh -n -f $USER_NAME@$HOST "sh -c 'rm -rf $GATLING_REPORT_DIR'"

done

for HOST in "${HOSTS[@]}"

echo "Copying simulations to host: $HOST"

scp -r $GATLING_SIMULATIONS_DIR/* $USER_NAME@$HOST:$GATLING_SIMULATIONS_DIR

done

for HOST in "${HOSTS[@]}"

echo "Running simulation on host: $HOST"

ssh -n -f $USER_NAME@$HOST "sh -c 'nohup $GATLING_RUNNER -nr -s $SIMULATION_NAME > /gatling/run.log 2>&1 &'"

done

echo "Running simulation on localhost"

$GATLING_RUNNER -nr -s $SIMULATION_NAME

echo "Gathering result file from localhost"

ls -t $GATLING_REPORT_DIR | head -n 1 | xargs -I {} mv ${GATLING_REPORT_DIR}{} ${GATLING_REPORT_DIR}report

cp ${GATLING_REPORT_DIR}report/simulation.log $GATHER_REPORTS_DIR

for HOST in "${HOSTS[@]}"

echo "Gathering result file from host: $HOST"

ssh -n -f $USER_NAME@$HOST "sh -c 'ls -t $GATLING_REPORT_DIR | head -n 1 | xargs -I {} mv ${GATLING_REPORT_DIR}{} ${GATLING_REPORT_DIR}report'"

scp $USER_NAME@$HOST:${GATLING_REPORT_DIR}report/simulation.log ${GATHER_REPORTS_DIR}simulation-$HOST.log

done

mv $GATHER_REPORTS_DIR $GATLING_REPORT_DIR

echo "Aggregating simulations"

$GATLING_RUNNER -ro reports

#using macOSX

open ${GATLING_REPORT_DIR}reports/index.html

#using ubuntu

#google-chrome ${GATLING_REPORT_DIR}reports/index.html

Gatling report example.

Stéphane LANDELLE

Do you mind if we add this to the documentation? Or even better, would you mind contributing it? https://github.com/gatling/gatling/blob/master/src/sphinx/cookbook/scaling_out.rst
- Nimrod007
  
  sure. will add it
- Nimrod007
  
  taking this 1 step further … created a image for a “Gatling Node”, deploying more nodes on demand when in need to run a “heavy” load test … will post the full code later on , for not I’ve updated the image template to use with this script
Anand Somani

Need more details..

Once ‘ ssh -n -f $USER_NAME@$HOST “sh -c ‘nohup $GATLING_RUNNER -nr -s $SIMULATION_NAME > /gatling/run.log 2>&1 &'”‘ command execute.

How script will get to know, execution of each server is completed?
- Nimrod007
  
  the script start the simulations on the remote hosts first, after it runs the simulation , after the simulation finished it takes all the results from the remote hosts (who probably finished before & you can add a sleep before that step) testing this always worked for me
Miki Lallouz ஜ

Hi guys, i made a script for “Auto Scaling Group” to automatically change the number of servers, wait for them to settle and run the script.

#set -o verbose
#set -x

echo -n “Number of servers > ”
read num

aws autoscaling update-auto-scaling-group –auto-scaling-group-name “LoadTest Group” –max-size $num –min-size $num –region us-east-1

running=-1

ip=wget http://ipinfo.io/ip -qO -

while [ $num -ne $running ]

do

HOSTS=””

count=0

running=0

for i in aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name "LoadTest Group" | grep -i instanceid | awk '{ print $2}' | cut -d',' -f1| sed -e 's/"//g'

do

t=aws ec2 describe-instances --instance-ids $i | grep -i PublicIp | awk '{ print $2 }' | head -1 | cut -d"," -f1| sed -e 's/"//g'

s=aws ec2 describe-instance-status --instance-ids $i | grep -i '"Status": "ok"' | awk '{ print $2 }' | head -1 | cut -d"," -f1| sed -e 's/"//g'

((count++))

if [ “$t” != “$ip” ]

then

if [ “$HOSTS” != “” ]

then

HOSTS=”$HOSTS ”

fi

HOSTS=”$HOSTS$t”

fi

if [ -n “$s” ]

then

((running++))

fi

done;

echo $HOSTS

echo “found $count machines”

echo “$running were fully running”

if [ $num -ne $running ]

then

echo “Sleeping 5s, then retry…”

echo “”

sleep 5s

fi

done

echo “——- All the servers are ready, executing script.”

HOSTS=($HOSTS)

— from here continue the script.
Miki Lallouz ஜ

BTW a problem i encountered. i had a problem where locally changes are being made but on remote ssh the commands doesnt seem to work. i managed to fix this by removing the “-f” from the ssh -n -f.
Thank you Nimrod!, helped me alot