Testing with CPload

Stein van Broekhoven - Cloud People

Why even build this tool?

Other tools were:

  • not easy enough
  • bloated with too many options I didn’t need
  • not easy to hack on (by me)

Features I was looking for

  • use access logs for traffic generation
  • change fqdn to be able to hit specific endpoints
  • gradual increase of traffic
  • rewrite specific requests
  • custom pre script (build session)
  • respect cookies / sessions / tokens etc.
  • high throughput per cpu core #lean

Use cases

  • During platform migration
  • In your application testing suite
  • Rebuilding or splitting applications
  • Testing new infra components
  • Testing auto scaling

Setup

git clone https://gitlab.com/cloud-people/cpload.git
cd cpload
mkvenv || pip install --user -r requirements.txt

Filters

They’re defined in a python file called filters.py you need 2 steps to add a filter

  1. Write a function that expands this basic one:

    def uri_editor(http_pool, url):
        send_request(http_pool, url)
    
  2. Add the function to the filter list

    filters = {
        '/magic/': magic_edit,
        'api/v1/date': future,
        'web': to_upper,
        '': send_request
    }
    

Filter example

Turn all urls containing api in to ALL CAPS

# ALL CAPS FOR ADDED DRAMA
def to_upper(http_pool, url):
    """URI TO UPPER"""
    send_request(http_pool, str(url).upper())

filters = {
    'api': to_upper,
    '': send_request
}
input='/api/v2/lalala/happy/url?data=test'
request='/API/V2/LALALA/HAPPY/URL?DATA=TEST'

What to feed this tool

  • Apache access logs
  • Nginx access logs
  • AWS ALB access logs
  • Hand crafted url list

A look at the options


optional arguments:
  -h, --help           show this help message and exit
  --urlfile URLFILE    the cleaned up http log
  --fromuri FROMURI    base uri/hostname as in the log file, this is used to edit it to the destination uri.
  --touri TOURI        the uri/hostname you want to send the traffic to
  --ratemin RATEMIN    minimum req/hour
  --ratemax RATEMAX    maximum req/hour
  --ramptime RAMPTIME  ramp up time in hours
  --duration DURATION  total test duration in hours
  --verbose            Output all request results
  --filters FILTERS    the uri filters and rewrites file

Now let’s fire it up