Aapje is Baas

PDF Network Printing Without Drivers

Tue, 18 Aug 2020 11:16:15 +0200

I’ve created a small docker container that exposes an API and a web interface to receive pdf files and forward them to an IPP printer on the network.

Dockerhub GitLab Source Code

If you like this project, please star it on GitLab and Dockerhub.

Why build something like this?

I needed an easy way for me as a human to print PDF files from all sorts of devices before I created this I used google CloudPrint which was great but they are shutting down the service. Also, it wasn’t flexible to be used with random scripts without jumping through complicated authentication hoops so I’ve created this simple http app that can run anywhere and point to any ipp printer it can reach. It uses cups to setup the connection to the printer and issue commands over IPP there is a simple check in the python server that verifies that it is a usable PDF and that the size is not too small with PyPDF2.

It is probably as plug and play as it gets, although I’m not entirely sure because I have only 1 network printer that I can test with.

How to use?

Run the container and place your printer’s ip in the env var IPP_IP like in this example docker run command:

docker run -d -p 5000:5000 -e "IPP_IP=192.168.1.66" aapjeisbaas/pdf-to-ipp

Now you can reach the web interface of the service at: http://127.0.0.1:5000/

Automate all the things!

This is pretty useful by itself but if you’re anything like me you want to automate your printing. The application expects A4 pdf files to be uploaded to: http://127.0.0.1:5000/uploader in a python function this will look like the one below.


import requests

def send_to_printer(pdf):
    files = {'file': open(pdf, 'rb')}
    requests.post('http://127.0.0.1:5000/uploader', files=files)

send_to_printer('/home/user/pdf/print-this.pdf')

It is cross-build to support all architectures I could think of and I run it on an ARM64 K3S cluster with raspberry pi’s and the container uses about 50MB of RAM.

This does not do X, Y OR Z

Nope, and probably never will.

This is mostly meant as a jumping-off point to “roll your own” and learn a bit about python/docker/cups/ipp along the way. It is not ultra configurable, nor do I need it to be. This thing just prints packing slips for my wife’s webshop and don’t want to overcomplicate things by adding to much “enterprise” sauce.

Do you have more printing projects?

Well, yes and it is even more niche than this one. This project is based on label-container and it takes in, you’ve probably guessed it already: PDF shipping labels, and it prints them on a Dymo 4XL

Problem-solving with lambda

Mon, 10 Aug 2020 16:20:13 +0200

My background is not predominantly programming I’ve spent most of my time as a system/infra/cloud/storage/networking engineer. Like most in my profession can tell you, we have 2 main tasks; build it & keep it online.

When you spend any time at a helpdesk type job and you’ll be well versed in solving a wide range of problems that customers face, that’s actually how this blog started. This makes you a creative and fast problem solver and if you’re lucky you’ll get enough edge cases you’ve never seen before to add to your arsenal of knowledge and keep yourself from going insane.

The takeaway from all of this is that you build muscle memory in fixing these problems.

Here’s my fix anything workflow:

Start with the point that was noticed by the client or monitoring
Figure out what delivered that thing that doesn’t work
Follow the break as deep as you need to go
Start fixing from there and follow all the way up
Repeat until you can verify the noticed thing to be working again.

Recently I realized that if you remove the deep dive into the root cause of the problem and replace it with a need for a solution and drill down to the core functionality you end up at the same place in the middle. From this point you then fix or build your way back up to the user-visible layers, this strategy is how I now build most of my lambda solutions.

“Start from the core problem and work out towards the user from there”

What is lambda?

AWS Lambda is an event-driven, serverless computing platform that aims to make it possible to deploy code in a smaller form factor and makes it possible to ship only a single function. The Lambda platform handles the interaction with the events and runs your code only when necessary, this makes it super simple to integrate with other components and it is only billed for the time your code is running. When your code is triggered it initializes a small container that runs your code, the items you have in memory that you initialize will stay available for a few minutes (as long as the container stays online) this enables you to make subsequent invocations faster. Lambda supports multiple languages I’ll be using python in this guide, for a full list of supported runtimes check out the lambda documentation.

Writing python lambda solutions

So let’s get to building some python code.

First up create a fresh virtualenv, you might want to use a Python virtualenv manager if you’re lazy like me use this: zsh-autoswitch-virtualenv This gives you a separate space for python packages only for this project.

1. virtualenv setup

# svanbroekhoven @ lynx in ~/git/tst [0:54:57] 
$ mkvenv
Creating tst virtualenv
created virtual environment CPython3.8.3.final.0-64 in 147ms
  creator CPython3Posix(dest=/home/svanbroekhoven/.virtualenvs/tst, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/svanbroekhoven/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Switching virtualenv: tst [🐍Python 3.8.3]
(tst) 
# svanbroekhoven @ lynx in ~/git/tst [0:55:03] 
$ cd ..
Deactivating: tst

# svanbroekhoven @ lynx in ~/git [1:53:20] 
$ cd tst          
Switching virtualenv: tst [🐍Python 3.8.3]
(tst) 
# svanbroekhoven @ lynx in ~/git/tst [1:53:27] 
$

As you can see in the example above, a virtualenv switcher helps if you’re working on multiple projects.

2. try to solve it once

Core problem: I need the nth Fibonacci number on many places where I don’t have much compute power or local storage to store enough of them.

Let’s open up a python shell or jupyter notebook and a web browser to try things and research possible solutions, in this example I’ll be using the example of a fibonacci number returning API. As a first project I would try to avoid complex dependencies outside of pure python, as a rough guide this means no Audio, Video and image-based projects. These libraries often require system-level dependencies which we can only install through building a lambda layer and that is a bit too involved for this guide. If you do want to dive into building layers check out this article in the lambda documentation.

The most basic Fibonacci function you can write looks like this, it has many problems but that leaves us some room for improvement in the future.

def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

for n in range(10 + 1):
    print(str(n)+  "\t" + str(fibonacci(n)))

3. Build and tweak your main.py

After some tinkering our basic script and function it looks like the one above, now my favorite way to proceed from here in put it in a main.py script and run it interactive like this: python -i main.py

# svanbroekhoven @ lynx in ~ [14:28:01] C:127
$ cd git/proza/content/post/problem-solving-with-lambda 
Switching virtualenv: problem-solving-with-lambda [Python 3.8.3]
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [14:28:16] 
$ cat main.py 
def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

for n in range(10 + 1):
    print(str(n)+  "\t" + str(fibonacci(n)))
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [14:28:59] 
$ python -i main.py
0	0
1	1
2	1
3	2
4	3
5	5
6	8
7	13
8	21
9	34
10	55
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'fibonacci', 'n']
>>> n
10   
>>> fibonacci(n)
55
>>> fibonacci(20)
6765
>>>

This way we can play with what we created without the need to paste our code in the interactive shell, this way we can hack on it and incorporate our first function in some other loops to validate if it’ll fit our requirements. Below I demonstrate my experiments in the interactive shell to try and show how I use this process.

# svanbroekhoven @ lynx in ~ [14:28:01] C:127
$ cd git/proza/content/post/problem-solving-with-lambda 
Switching virtualenv: problem-solving-with-lambda [Python 3.8.3]
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [14:28:16] 
$ cat main.py 
def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

for n in range(10 + 1):
    print(str(n)+  "\t" + str(fibonacci(n)))
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [14:28:59] 
$ python -i main.py
0	0
1	1
2	1
3	2
4	3
5	5
6	8
7	13
8	21
9	34
10	55
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'fibonacci', 'n']
>>> n
10
>>> fibonacci(n)
55

That was the fix once side over now let’s do it in a loop and display the input and output to the user. You can read all the KeyboardInterrupt as the operation took too long, I killed it to keep working instead of waiting on it.

>>> for n in range(10, 200):
...   fibonacci(n)
... 
55
89
144
 ...
 ...
 ...
5702887
9227465
^C
KeyboardInterrupt
>>> for n in range(10, 200):
...   print( str(n) + ": " + str(fibonacci(n)))
... 
10: 55
11: 89
 ...
 ...
 ...
36: 14930352
37: 24157817
^C
KeyboardInterrupt
>>> import random
>>> random.randrange(10)
6
>>> 
>>> 
>>> for i in range(10):
...   n = random.randrange(20)
...   print( str(n) + ": " + str(fibonacci(n)))
... 
3: 2
5: 5
5: 5
12: 144
15: 610
1: 1
1: 1
13: 233
4: 3
11: 89
>>> for i in range(10):
...   n = random.randrange(200)
...   print( str(n) + ": " + str(fibonacci(n)))
... 

^C
KeyboardInterrupt
>>> from datetime import datetime, timedelta 
>>> now = datetime.now()
>>> start = datetime.now()
>>> delta = datetime.now() - start 
>>> delta
datetime.timedelta(seconds=37, microseconds=994986)
>>> print(delta)
0:00:37.994986
>>> for i in range(10):
...   start = datetime.now()
...   n = random.randrange(50)
...   print( str(n) + ": " + str(fibonacci(n)) + " " + str(datetime.now() - start))
... 
41: 165580141 0:00:43.840478
3: 2 0:00:00.000009
9: 34 0:00:00.000011
^C
KeyboardInterrupt
>>>

What I’ve accomplished this far is a simple loop that prints out 10 random fib numbers in a range from 0 to 50 and print time taken to complete it, but the results are really slow so I needed a built-in stopwatch to do some crude benchmarking of the function. I now start with placing some of my experiments back into the main.py with vim and make sure this “benchmark” auto starts as I run the file. I’ll add a lru_cache to the function to cache results to hide the slowness and make the output a bit more readable.

(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [15:06:00] 
$ vim main.py      
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [15:06:06] 
$ python -i main.py
38: 39088169 0:00:00.000024
14: 377 0:00:00.000002
10: 55 0:00:00.000002
26: 121393 0:00:00.000001
17: 1597 0:00:00.000002
29: 514229 0:00:00.000001
30: 832040 0:00:00.000001
46: 1836311903 0:00:00.000005
4: 3 0:00:00.000001
28: 317811 0:00:00.000001
>>> 
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [15:06:23] 
$ vim main.py      
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [15:07:45] 
$ python -i main.py
11	0:00:00.000005 89
4	0:00:00.000002 3
17	0:00:00.000001 1597
30	0:00:00.000001 832040
47	0:00:00.000001 2971215073
38	0:00:00.000001 39088169
35	0:00:00.000002 9227465
44	0:00:00.000001 701408733
2	0:00:00.000001 1
1	0:00:00.000001 1
>>> 
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [15:08:05] 
$ vim main.py      
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [15:08:17] 
$ python -i main.py
67	0:00:00.000007 44945570212853
20	0:00:00.000004 6765
72	0:00:00.000002 498454011879264
79	0:00:00.000001 14472334024676221
27	0:00:00.000002 196418
68	0:00:00.000002 72723460248141
33	0:00:00.000002 3524578
10	0:00:00.000002 55
17	0:00:00.000002 1597
90	0:00:00.000001 2880067194370816120
>>>

Well this looks kinda oke, let’s get rid of the auto run benchmark crap and add a function that triggers the benchmark with a range variable.

(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [15:08:23] 
$ vim main.py      
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [15:09:33] 
$ python -i main.py
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'datetime', 'fibonacci', 'lru_cache', 'random', 'range_test']
>>> range_test(100)
61	0:00:00.000066 2504730781961
50	0:00:00.000017 12586269025
61	0:00:00.000010 2504730781961
93	0:00:00.000010 12200160415121876738
2	0:00:00.000015 1
17	0:00:00.000011 1597
82	0:00:00.000011 61305790721611591
15	0:00:00.000011 610
65	0:00:00.000011 17167680177565
61	0:00:00.000006 2504730781961
>>> range_test(500)
262	0:00:00.000029 2542592393026885507715496646813780220945054040571721231
468	0:00:00.000016 28624020537229717283244863695841661789309459371299941322398704536965075971940465647510004531000816
424	0:00:00.000011 18250487254938611555074410636524312658020849229014745928158881386149462839394269270468043
237	0:00:00.000006 15156039800290547036315704478931467953361427680642
496	0:00:00.000006 20341574322680408081083829243820203612317308197211964554628215486203974898255803242740333222721700974747
402	0:00:00.000014 460835978753503578226215883073872246385764472086797082873203188542544616448248343576
286	0:00:00.000006 263621064469290555679241849789653324393054271110084140201023
387	0:00:00.000006 337856107814181089864066841370437948648180483483258553487263501935155470092051458
24	0:00:00.000005 46368
371	0:00:00.000005 153083904475345790698149223310665389766178449653686710164582374234640876900329
>>> 
(problem-solving-with-lambda) 
# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [15:10:09] 
$ cat main.py 
from functools import lru_cache
from datetime import datetime
import random

@lru_cache(maxsize = None)
def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

def range_test(nMax):
    for i in range(10):
      start = datetime.now()
      n = random.randrange(nMax)
      print( str(n) + "\t" + str(datetime.now() - start) + " " + str(fibonacci(n)))

I discovered that it was way to slow and by adding a lru_cache to the function it became a bit more responsive and usable for our purpose. Also I’ve changed the old range test a new variable nMax version with random numbers and a time spent counter. Now for a final cherry on top let’s add a main, if you’re familiar with other programming languages this is probably what you’re used to but in python it is not strictly necessary.

if __name__ == "__main__":
    # execute only if run as a script
    main()

The code above will only be triggered if the script is executed directly and not when the file is imported. Let’s adapt it to our situation and run a simple benchmark when started directly as a script. The main snippet is always placed at the bottom of our script as all the functions need to placed in memory before we can call them in our main.

from functools import lru_cache
from datetime import datetime
import random

@lru_cache(maxsize = None)
def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

def range_test(nMax):
    for i in range(10):
      start = datetime.now()
      n = random.randrange(nMax)
      print( str(n) + "\t" + str(datetime.now() - start) + " " + str(fibonacci(n)))

if __name__ == "__main__":
    # execute only if run as a script
    range_test(100)

This wil give us the following result:

# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [10:35:55] 
$ python main.py 
96      0:00:00.000005 51680708854858323072
58      0:00:00.000003 591286729879
53      0:00:00.000001 53316291173
0       0:00:00.000001 0
12      0:00:00.000001 144
45      0:00:00.000001 1134903170
32      0:00:00.000001 2178309
73      0:00:00.000001 806515533049393
97      0:00:00.000001 83621143489848422977
44      0:00:00.000001 701408733

# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [10:35:59] 
$

Now this isn’t very useful, let’s add some cli params to make it calculate a Fibonacci number.

if __name__ == "__main__":
    # User input handling
    import argparse
    parser = argparse.ArgumentParser(description="Returns the n'th fibonacci number.")
    parser.add_argument('-n', required=True, type=int, help="which number in the fibonacci sequence you want")
    args = parser.parse_args()

    # execute calculation
    print(fibonacci(args.n))

# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [10:44:07] 
$ python main.py        
usage: main.py [-h] -n N
main.py: error: the following arguments are required: -n

# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [10:44:34] C:2
$ python main.py -n 20 
6765

# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [10:47:35] 
$ python main.py -n 200
280571172992510140037611932413038677189525

That’s yielding us a lot more value and a pleasant help text if we forget to pass a -n #num, now let’s add our calculation part of the main block to a lambda handler function to give lambda a way to interact with it and add a api gateway test event to test the lambda_handler function. I won’t go into details about this if you want to read more just follow the links below, for now we only care about the following location in the event: queryStringParameters All data is passed through in JSON so we will need to import json at the top of our file.

I use the API Gateway implementation here, but there are many ways to trigger and each has it’s own event body please check the documentation to find out how to use AWS Lambda with other services.

def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    return fibonacci(int(event['queryStringParameters']['n']))

# This represents an api gateway request to: https://sub.domain.tld/api/v1/fibonacci?n=100
test_event = json.loads("""{
  "path": "/api/v1/fibonacci",
  "httpMethod": "GET",
  "queryStringParameters": {
    "n": "100"
  },
  "pathParameters": {
    "proxy": "/fibonacci"
  },
  "requestContext": {
    "identity": {
      "sourceIp": "127.0.0.1",
      "userAgent": "Custom User Agent String"
    },
    "path": "/prod/fibonacci",
    "protocol": "HTTP/1.1"
  }
}""")

# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [11:24:40] 
$ python -i main.py -n 100
354224848179261915075
>>> lambda_handler(test_event, '')
354224848179261915075
>>> fibonacci(100)
354224848179261915075
>>> 

# svanbroekhoven @ lynx in ~/git/proza/content/post/problem-solving-with-lambda on git:master x [11:25:21] 
$

Now as a final step to complete our local lambda test we run our lambda handler from our main block with the test event, I also place the test event into the main block to reduce memory usage in lambda.

from functools import lru_cache
from datetime import datetime
import random
import json

@lru_cache(maxsize = None)
def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

def range_test(nMax):
    for i in range(10):
      start = datetime.now()
      n = random.randrange(nMax)
      print( str(n) + "\t" + str(datetime.now() - start) + " " + str(fibonacci(n)))

def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    return fibonacci(int(event['queryStringParameters']['n']))


if __name__ == "__main__":
    # User input handling
    import argparse
    parser = argparse.ArgumentParser(description="Returns the n'th fibonacci number.")
    parser.add_argument('-n', required=True, type=int, help="which number in the fibonacci sequence you want")
    args = parser.parse_args()

    # execute calculation directly
    # print(fibonacci(args.n))

    # This represents an api gateway request to: https://sub.domain.tld/api/v1/fibonacci?n=100
    test_event = json.loads("""{
      "path": "/api/v1/fibonacci",
      "httpMethod": "GET",
      "queryStringParameters": {
        "n": "100"
      },
      "pathParameters": {
        "proxy": "/fibonacci"
      },
      "requestContext": {
        "identity": {
          "sourceIp": "127.0.0.1",
          "userAgent": "Custom User Agent String"
        },
        "path": "/prod/fibonacci",
        "protocol": "HTTP/1.1"
      }
    }""")

    # insert aur desired n in the test_event
    test_event['queryStringParameters']['n'] = args.n
    # execute calculation with test event through lambda_handler function
    print(lambda_handler(test_event, ''))

Packaging python lambda functions

To prepare our function to be able to run in Lambda we need to zip it together with it’s dependencies this sounds pretty easy, and guess what… it is! I use GitLab for remote git storage and their great CI/CD pipelines, my examples will be based around GitLab but you can use whatever you’re comfortable with.

There are several ways to deploy your function:

1. AWS Web Console

This is only usable if you have no external dependencies and small scripts, it is user friendly to setup but updating code will become a fragile human-driven process.

2. AWS Lambda CLI

This is often used in examples from AWS themselves, it is a lot better than copying and pasting in the web console but still not what I’d recommend for production use. The syntax is quite clear and the reference helps you to do everything you can think of with the lambda service.

# AWS Lambda CLI update code example
aws lambda update-function-code \
    --function-name  my-function \
    --zip-file fileb://my-function.zip

3. AWS Cloudformation

This is my favorite way of deploying on AWS, it gives you full control over the recourses and makes expanding with other services like DynamoDB at a later point in time much easier since you’ve already build your infra as code pipeline. This is also the way we’ll be deploying the code in this tutorial.

4. AWS Serverless Application Model

From the brochure: Build serverless applications in simple and clean syntax

To me this wasn’t a super intuitive method and more confusing because it isn’t just plain vanilla cloudformation and multi-account deployment isn’t supported out of the box. I did make a multi-account hack on top of sam but I wouldn’t recommend it. If you’re still curious here is the AWS SAM Overview page and a blank python sam project

CI/CD Packaging and deployment with Gitlab and Cloudformation

Before we take it apart step by step, this is what our Gitlab pipeline and cfn.yml (cloudformation) will look like.

default:
  image: python

stages:
  - deployment

deployment:
  only:
    refs:
      - master
  stage: deployment
  artifacts:
    paths:
    - fibonacciPackage.zip
    expire_in: 1 week
  script: 
    - apt update -qqq && apt install -qqq -y zip > /dev/null # pull in zip as it is not in the python default container
    - pip install --quiet awscli                             # need this for the upload but not in the package
    - pip install --quiet -r requirements.txt -t ./          # silently install requirements in the current dir
    - chmod -R 755 .
    - zip -r fibonacciPackage.zip . -x ".git*" --quiet
    - export zipname="fibonacciPackage-$(date +%s).zip"
    - aws s3 cp fibonacciPackage.zip s3://bucket/lambda-packages/fibonacci/${zipname}
    - aws cloudformation deploy --template-file cfn.yml --capabilities CAPABILITY_IAM --stack-name fibonacci --no-fail-on-empty-changeset --parameter-overrides s3Bucket="bucketname" s3Key="lambda-packages/fibonacci/${zipname}"

AWSTemplateFormatVersion: '2010-09-09'
Description: fibonacci service

Parameters:
  s3Bucket:
    Type: String
    Default: "bucket"
  s3Key:
    Type: String
    Default: "lambda-packages/fibonacci/fibonacciPackage.zip"

Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: root
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - logs:*
            Resource: arn:aws:logs:*:*:*

  FibonacciFunction: 
    Type: AWS::Lambda::Function
    Properties:
      Handler: main.lambda_handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code: 
        S3Bucket: !Ref s3Bucket
        S3Key: !Ref s3Key
      Runtime: 'python3.8'
      Timeout: 20
      MemorySize: 128

Steps required for python lambda packaging

Basically there are 2 steps:

install your requirements locally
zip it

install requirements in the local directory

pip makes this really easy, you can install multiple packages with a requirements.txt file or a single one directly.

Without requirements.txt

pip install boto3 -t ./
pip install requests -t ./
pip install git+https://github.com/psf/requests.git -t ./

With requirements.txt

pip install -r requirements.txt -t ./

If you need python packages in your pipeline but don’t want to include them in you lambda package just use pip without the -t parameter. Also If you don’t like the amount of logs pip creates add a --quiet parameter to silence the noisy output.

pip install --quiet awscli

zip it

Now zip your project, change the permissions to prevent issues surrounding that and exclude things you won’t need in runtime.

chmod -R 755 . # makes sure there are no permission errors 
zip -r lambdaPackage.zip . -x ".git*" --quiet

Upload to S3

Deployment to lambda is easiest if you can point to your deployment zip in S3, to do this you could take a couple of naming roads I’ve used all of them, and technically there isn’t a clear best way to do it in my opinion. You shouldn’t overwrite your old zip file when releasing a new version, this makes rollbacks only possible if you also rebuild your old version and that’s not the desired behavior.

Here are a few of my favorite zip identifiers:

git short hash git rev-parse --short HEAD
epoch date date +%s
random string openssl rand -hex 12
sem ver of the app git tag --points-at HEAD (untested)
build number > check your pipeline manual

For now, I’ll use epoch timestamped zip files, but you could replace the date +%s with whatever command you prefer to tag your lambda packages.

zip -r lambdaPackage.zip . -x ".git*" --quiet
export zipname="lambdaPackage-$(date +%s).zip"
aws s3 cp lambdaPackage.zip s3://bucket/lambda-packages/projectName/${zipname}

After those commands, you can use the env var zipname in the following deployment to point to your new zip file.

Deploy with cloudformation

A deep dive into cloudformation is beyond the scope here so I’ll just give you my default basic deploy command:

aws cloudformation deploy --template-file cfn.yml --capabilities CAPABILITY_IAM --stack-name fibonacci --no-fail-on-empty-changeset --parameter-overrides s3Bucket="bucketname" s3Key="lambda-packages/projectName/${zipname}"

Now you should be able to see your lambda in your AWS account, if not there are a couple of things that could have gone wrong. No panic at the bottom of this article I’ve listed some common problems you can have with this kind of setup and how to debug and fix them.

Testing, improving, and finalizing your lambda in aws

Now that your Lambda is ready to be used, have a go with the build-in testing in the web interface. On the Lambda function page at the to you’ll find a bar with a test button next to it from the dropdown click Configure test event, here you can input the json from your script or try out different examples. This test event in the web interface is my preferred way of trying new things in lambda because I have all the logs and metrics right in front of me. If you’re looking for a more hands-off automated test system approach have a look at the aws lambda invoke cli reference.

From here, just repeat the following: tweak, commit, push, test, and once your results are to your likings connect it to your preferred trigger and start testing the whole stack. Keep in mind that triggers will need iam permissions to trigger your lambda.

Help it’s not working for me

Here’s a checklist to fix most of the common issues with automated deployments like this one.

aws command not found

pip install --quiet awscli

Are there AWS credentials attached to the pipeline?

aws sts get-caller-identity

Deployment / upload fails

Do your iam credentials connected to the pipeline allow you to use: cloudformation, lambda, logs, and S3? Did you select the correct bucket name and is that bucket already created, if not create it before you deploy.

OSError: no library called … was found

[ERROR] OSError: no library called "cairo" was found no library called "libcairo-2" was found cannot load library 'libcairo.so': libexpat.so.1: cannot open shared object file: No such file or directory cannot load library 'libcairo.2.dylib': libcairo.2.dylib: cannot open shared object file: No such file or directory....

You’re trying to call an os lib that is not installed, you can fix this by building a lambda layer that provides the necessary lib files. A lambda layer is another zip file that provides some files to the container runtime a good place to start when building a lambda layer is the AWS Lambda layers page in the lambda configuration documentation. If you want to have a look at the solution for the layer solution to the above problem, check out cloud-print-utils.

Fibonacci example project on gitlab

Popular Movies

Mon, 22 Jun 2020 15:03:34 +0200

An api to pull in top popular movies from imdb

This api was inspired by Steven Lu’s popular-movies project.

I wanted to follow and filter the imdb moviemeter list but couldn’t find a direct api or a list from Steven Lu that reflected what I was looking for. So that’s where this project comes in. It is written in python and runs in AWS as a lambda fuction triggerd by an api gateway which sits behind a cloudfront to cache the majority of the requests. Within the function itself there’s also caching built in to prevent unnecessary calls to imdb and speed up responses to users if the data is already in memory.

If you want to see the code behind this api have a look at the gitlab project page.

options

Query Param	Effect	value ranges
fresh	Only return current and last year releases	true / false
year	Return $year and newer (disabled with fresh)	4 number year
rating	Return $rating and higher	0 - 10
votes	Return movies with at least this many votes	1 - ∞
max	Max returned movies	1 - ∞
list	Select a imdb chart source default=popular	popular / top

Example Queries:

curl | jq

$ curl -s -X GET 'https://aapjeisbaas.nl/api/v1/popular-movies/imdb?fresh=True&max=3&rating=6&votes=50000&list=popular' |jq
[
  {
    "title": "Extraction",
    "imdb_id": "tt8936646",
    "rating": 6.8,
    "year": 2020,
    "votes": 101076,
    "popular movies 100 rank": 1
  },
  {
    "title": "Star Wars: Episode IX - The Rise of Skywalker",
    "imdb_id": "tt2527338",
    "rating": 6.7,
    "year": 2019,
    "votes": 319690,
    "popular movies 100 rank": 2
  },
  {
    "title": "Once Upon a Time... in Hollywood",
    "imdb_id": "tt7131622",
    "rating": 7.7,
    "year": 2019,
    "votes": 462778,
    "popular movies 100 rank": 6
  }
]

Charts you can query

list param	IMDb link	api example
top	https://www.imdb.com/chart/top/	https://aapjeisbaas.nl/api/v1/popular-movies/imdb?fresh=True&list=top
popular	https://www.imdb.com/chart/moviemeter/	https://aapjeisbaas.nl/api/v1/popular-movies/imdb?fresh=True&list=popular

Radarr

This list by itself isn’t all that exiting, the magic happens when you use it in Radarr as a list source. This api uses a structure that is compatible with the StevenLuImport so you can just input the query in the URL field.

Please use at a sensible rate of max 1 query per hour in your radar setup.

There seems to be an issue with the auto library clean future and StevenLu lists, so if you use this list you need to set: Clean Library Level to: Log only There are already multiple github issues open to solve this, unfortunatley this is not in the regular releases yet. (In the aphrodite branch this is fixed, but trust me it is unstable and you don’t want to run it.) issue 4014 pull req

The api in action

This table is created using the following url:

https://aapjeisbaas.nl/api/v1/popular-movies/imdb?fresh=True&max=20&rating=6&votes=50000

Movie Title	Rating	Year	Votes	IMDb

Donate

If you like this api and want to give something back please donate to keep it around and up to date.

method	where
XRP	rPThf2QWMKKkmQM8hw9yD4bwpbmLYiz6vB
ETH	0xe589531E35fA84d9d561f028070663004aca5Fa4
BCH	bitcoincash:qr5t82vekd7ekx7hsfkdqlwsws67nq3dcg6hn92u8x
BTC	15aaSnoGBUMxyGwt7JveSNT75hBCL14wu1
iDeal	https://bunq.me/svb/_/movies-api

Rocksmith on Linux

Tue, 10 Mar 2020 23:52:17 +0100

Rocksmith is a game / guitar teacher for Windows and game consoles that allows you to use a real guitar to play minigames, learn actual songs and overall improve your guitar skills. I’ve been playing this off and on since the original game out in late 2012 with 1 big frustration aside from the playing guitar is hard part. It never had support for linux and when I got it to work I had problems with stability and a terribly complex setup.

I have no latency issues because I don’t try to use my PC as a software amp like guitarix and have the audio output in the game turned off. This doesn’t change the gameplay at all, I play with my headphone on one ear so I can hear my guitar with the other.

So let’s fix it!

To replicate my setup you need:

Electric guitar
Real tone cable or any dedicated USB guitar line in device
Linux desktop setup, (Arch / Fedora / Ubuntu)

Install Steam and Rocksmith

I used flatpak and it is super stable for me if you dont have flatpak installed follow your distro’s install guide here

flatpak install flathub com.valvesoftware.Steam

After the installation is complete go in to the setting and enable Steam Play for all titles. (I tested with proton 3.16-9 but newer versions will probably work fine)

Now go ahead and install Rocksmith and see if your guitar works instantly, if so have fun.

Fix audio input

Darn it no luck, but don’t worry we’ll get there.

I created a /usr/local/bin/steam with a disable pulse for wine, better home mapping and auto update on startup. You will get this to work with only the export and the flatpak run but this is just how I like to start my steam client.

export WINENOPULSE=1
sudo flatpak update com.valvesoftware.Steam -y
sudo flatpak override com.valvesoftware.Steam --filesystem=$HOME
flatpak run com.valvesoftware.Steam

Find your steam library path mine is in /home/svanbroekhoven/steam_lib/ so replace that with your path in the examples below.

To edit the wine config / registry files I used protontricks and this can also be installed using a flatpak

[svanbroekhoven@alan ~]$ flatpak install flathub com.valvesoftware.Steam.Utility.protontricks
Looking for matches…


        ID                                             Branch    Op    Remote     Download
 1. [✓] com.valvesoftware.Steam.Utility.protontricks   stable    i     flathub    2.7 MB / 3.2 MB

Installation complete.

Now check if your protontricks can find the installed games:

[svanbroekhoven@alan ~]$ alias protontricks-flat='flatpak run --command=protontricks com.valvesoftware.Steam --no-runtime'
[svanbroekhoven@alan ~]$ protontricks-flat -s rocksmith
Found the following games:
Rocksmith® 2014 Edition - Remastered (221680)

To run protontricks for the chosen game, run:
$ protontricks APPID COMMAND

NOTE: A game must be launched at least once before protontricks can find the game.
[svanbroekhoven@alan ~]$

Now let’s change the sound system to alsa for this game:

[svanbroekhoven@alan ~]$ protontricks-flat 221680 sound=alsa
------------------------------------------------------
Your version of wine 3.16 is no longer supported upstream. You should upgrade to 4.x
------------------------------------------------------
------------------------------------------------------
You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
------------------------------------------------------
Using winetricks 20191224 - sha256sum: afe039a7d72553cb761f0367f9f2085b92af8caf86e025c34bbc1fdd89a1f9ee with wine-3.16 and WINEARCH=win64
Executing w_do_call sound=alsa
------------------------------------------------------
You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
------------------------------------------------------
Executing load_sound alsa
Setting sound driver to alsa
Executing /home/svanbroekhoven/steam_lib/steamapps/common/Proton 3.16/dist/bin/wine regedit C:\windows\Temp\set-sound.reg
wine: cannot find L"C:\\windows\\system32\\winemenubuilder.exe"
000b:err:wineboot:ProcessRunKeys Error running cmd L"C:\\windows\\system32\\winemenubuilder.exe -a -r" (2)
0021:err:module:load_builtin_dll failed to load .so lib for builtin L"winebus.sys": libudev.so.0: cannot open shared object file: No such file or directory
0021:err:ntoskrnl:ZwLoadDriver failed to create driver L"\\Registry\\Machine\\System\\CurrentControlSet\\Services\\WineBus": c0000142
000f:fixme:service:scmdatabase_autostart_services Auto-start service L"WineBus" failed to start: 1114
0014:err:service:process_send_command service protocol error - failed to write pipe!
Executing /home/svanbroekhoven/steam_lib/steamapps/common/Proton 3.16/dist/bin/wine64 regedit C:\windows\Temp\set-sound.reg
wine: cannot find L"C:\\windows\\system32\\winemenubuilder.exe"
000b:err:wineboot:ProcessRunKeys Error running cmd L"C:\\windows\\system32\\winemenubuilder.exe -a -r" (2)
0021:err:module:load_builtin_dll failed to load .so lib for builtin L"winebus.sys": libudev.so.0: cannot open shared object file: No such file or directory
0021:err:ntoskrnl:ZwLoadDriver failed to create driver L"\\Registry\\Machine\\System\\CurrentControlSet\\Services\\WineBus": c0000142
0014:err:service:process_send_command service protocol error - failed to write pipe!
000f:fixme:service:scmdatabase_autostart_services Auto-start service L"WineBus" failed to start: 1114
[svanbroekhoven@alan ~]$

This output looks kinda broken but it works for me so….

The last command edited our .reg files in /home/svanbroekhoven/steam_lib/steamapps/compatdata/221680/pfx/ and the most notable change is this:

cat /home/svanbroekhoven/steam_lib/steamapps/compatdata/221680/pfx/system.reg | grep 'Pulse' -B 5 -A 5  
#time=1d5f7264ac701fc
"DeviceState"=dword:00000004

[Software\\Microsoft\\Windows\\CurrentVersion\\MMDevices\\Audio\\Capture\\{25DA76D0-033C-4235-9002-19F48894AC6F}\\Properties] 1583876540
#time=1d5f724c70e68ec
"{026E516E-B814-414B-83CD-856D6FEF4822},2"="Pulseaudio"
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},0"=dword:00000004
"{233164C8-1B2C-4C7D-BC68-B671687A2567},1"="{25DA76D0-033C-4235-9002-19F48894AC6F}"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},14"="Pulseaudio"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},2"="Pulseaudio"
"{E4870E26-3CC5-4CD2-BA46-CA0A9A70ED04},3"=hex:fe,ff,01,00,44,ac,00,00,10,b1,\
  02,00,04,00,20,00,16,00,20,00,04,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
  aa,00,38,9b,71
"{F19F064D-082C-4E27-BC73-6882A1BB8E4C},0"=hex:fe,ff,01,00,44,ac,00,00,10,b1,\
  02,00,04,00,20,00,16,00,20,00,04,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
--
#time=1d5f7264ac7137c
"DeviceState"=dword:00000004

[Software\\Microsoft\\Windows\\CurrentVersion\\MMDevices\\Audio\\Render\\{FD47D9CC-4218-4135-9CE2-0C195C87405B}\\Properties] 1583876540
#time=1d5f724c70e5848
"{026E516E-B814-414B-83CD-856D6FEF4822},2"="Pulseaudio"
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},0"=dword:00000001
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},3"=dword:00000003
"{233164C8-1B2C-4C7D-BC68-B671687A2567},1"="{FD47D9CC-4218-4135-9CE2-0C195C87405B}"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},14"="Pulseaudio"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},2"="Pulseaudio"
"{E4870E26-3CC5-4CD2-BA46-CA0A9A70ED04},3"=hex:fe,ff,02,00,44,ac,00,00,20,62,\
  05,00,08,00,20,00,16,00,20,00,03,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
  aa,00,38,9b,71
"{F19F064D-082C-4E27-BC73-6882A1BB8E4C},0"=hex:fe,ff,02,00,44,ac,00,00,20,62,\
  05,00,08,00,20,00,16,00,20,00,03,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\

Now for out final step use winecfg to manualy set the mic and line input to our sound input card. For that you first select the correct wineprefix.

You might need to install wine on your system yo be able to run winecfg.

sudo pacman -S wine

# deafult flatpak location
export WINEPREFIX="/home/svanbroekhoven/.var/app/com.valvesoftware.Steam/data/Steam/steamapps/compatdata/221680/pfx"
# custom steam library location
export WINEPREFIX="/home/svanbroekhoven/steam_lib/steamapps/compatdata/221680/pfx"
winecfg
0029:err:setupapi:create_dest_file failed to create L"C:\\windows\\system32\\atl100.dll" (error=80)
0029:err:winediag:SECUR32_initNTLMSP ntlm_auth was not found or is outdated. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.
0029:fixme:dwmapi:DwmIsCompositionEnabled 000000006DFA0434
002e:fixme:iphlpapi:NotifyIpInterfaceChange (family 0, callback 0x69ed306d, context 0x421060, init_notify 0, handle 0x182fa00): stub
0029:fixme:ntdll:NtQuerySystemInformation info_class SYSTEM_PERFORMANCE_INFORMATION
0040:err:setupapi:create_dest_file failed to create L"C:\\windows\\system32\\atl100.dll" (error=80)
0040:err:winediag:gnutls_initialize failed to load libgnutls, no support for encryption
0040:err:winediag:gnutls_initialize failed to load libgnutls, no support for pfx import/export
0040:err:module:load_so_dll failed to load .so lib "/usr/bin/../lib32/wine/mp3dmod.dll.so": libmpg123.so.0: cannot open shared object file: No such file or directory
0040:err:winediag:schan_imp_init Failed to load libgnutls, secure connections will not be available.
0040:err:winediag:SECUR32_initNTLMSP ntlm_auth was not found or is outdated. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.
0040:err:winediag:load_gssapi_krb5 Failed to load libgssapi_krb5, Kerberos SSP support will not be available.
0040:fixme:dwmapi:DwmIsCompositionEnabled 6D95DD14
0042:fixme:iphlpapi:NotifyIpInterfaceChange (family 0, callback 0x6a0df537, context 0x3af348, init_notify 0, handle 0x176fce8): stub
0040:fixme:ntdll:NtQuerySystemInformation info_class SYSTEM_PERFORMANCE_INFORMATION
0040:err:module:load_so_dll failed to load .so lib "/usr/bin/../lib32/wine/winegstreamer.dll.so": libgstvideo-1.0.so.0: cannot open shared object file: No such file or directory
wine: configuration in L"/home/svanbroekhoven/steam_lib/steamapps/compatdata/221680/pfx" has been updated.

Accept all the install crap from wine

wait…

Go to the Audio tab, select Rocksmith cable for input and apply the changes.

Close the winecfg app and start your game.

If it’s still not perfect have a look at /home/svanbroekhoven/steam_lib/steamapps/common/Rocksmith2014/Rocksmith.ini

[Audio]
EnableMicrophone=1
ExclusiveMode=0
LatencyBuffer=4
ForceDefaultPlaybackDevice=
ForceWDM=0
ForceDirectXSink=0
DumpAudioLog=0
MaxOutputBufferSize=0
RealToneCableOnly=0
Win32UltraLowLatencyMode=0
[Renderer.Win32]
ShowGamepadUI=0
ScreenWidth=0
ScreenHeight=0
Fullscreen=1
VisualQuality=2
RenderingWidth=0
RenderingHeight=0
EnablePostEffects=1
EnableShadows=1
EnableHighResScope=1
EnableDepthOfField=1
EnablePerPixelLighting=1
MsaaSamples=4
DisableBrowser=0
[Net]
UseProxy=1

And just to be as complete as possible here are some snippets from the system.reg

[svanbroekhoven@alan ~]$ cat /home/svanbroekhoven/steam_lib/steamapps/compatdata/221680/pfx/system.reg | grep 'Pulse' -B 5 -A 5
#time=1d5f7264ac701fc
"DeviceState"=dword:00000004

[Software\\Microsoft\\Windows\\CurrentVersion\\MMDevices\\Audio\\Capture\\{25DA76D0-033C-4235-9002-19F48894AC6F}\\Properties] 1583876540
#time=1d5f724c70e68ec
"{026E516E-B814-414B-83CD-856D6FEF4822},2"="Pulseaudio"
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},0"=dword:00000004
"{233164C8-1B2C-4C7D-BC68-B671687A2567},1"="{25DA76D0-033C-4235-9002-19F48894AC6F}"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},14"="Pulseaudio"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},2"="Pulseaudio"
"{E4870E26-3CC5-4CD2-BA46-CA0A9A70ED04},3"=hex:fe,ff,01,00,44,ac,00,00,10,b1,\
  02,00,04,00,20,00,16,00,20,00,04,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
  aa,00,38,9b,71
"{F19F064D-082C-4E27-BC73-6882A1BB8E4C},0"=hex:fe,ff,01,00,44,ac,00,00,10,b1,\
  02,00,04,00,20,00,16,00,20,00,04,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
--
#time=1d5f7264ac7137c
"DeviceState"=dword:00000004

[Software\\Microsoft\\Windows\\CurrentVersion\\MMDevices\\Audio\\Render\\{FD47D9CC-4218-4135-9CE2-0C195C87405B}\\Properties] 1583876540
#time=1d5f724c70e5848
"{026E516E-B814-414B-83CD-856D6FEF4822},2"="Pulseaudio"
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},0"=dword:00000001
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},3"=dword:00000003
"{233164C8-1B2C-4C7D-BC68-B671687A2567},1"="{FD47D9CC-4218-4135-9CE2-0C195C87405B}"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},14"="Pulseaudio"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},2"="Pulseaudio"
"{E4870E26-3CC5-4CD2-BA46-CA0A9A70ED04},3"=hex:fe,ff,02,00,44,ac,00,00,20,62,\
  05,00,08,00,20,00,16,00,20,00,03,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
  aa,00,38,9b,71
"{F19F064D-082C-4E27-BC73-6882A1BB8E4C},0"=hex:fe,ff,02,00,44,ac,00,00,20,62,\
  05,00,08,00,20,00,16,00,20,00,03,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
[svanbroekhoven@alan ~]$ cat /home/svanbroekhoven/steam_lib/steamapps/compatdata/221680/pfx/system.reg | grep 'Pulse\|Rock' -B 5 -A 5
#time=1d5f7264ac701fc
"DeviceState"=dword:00000004

[Software\\Microsoft\\Windows\\CurrentVersion\\MMDevices\\Audio\\Capture\\{25DA76D0-033C-4235-9002-19F48894AC6F}\\Properties] 1583876540
#time=1d5f724c70e68ec
"{026E516E-B814-414B-83CD-856D6FEF4822},2"="Pulseaudio"
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},0"=dword:00000004
"{233164C8-1B2C-4C7D-BC68-B671687A2567},1"="{25DA76D0-033C-4235-9002-19F48894AC6F}"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},14"="Pulseaudio"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},2"="Pulseaudio"
"{E4870E26-3CC5-4CD2-BA46-CA0A9A70ED04},3"=hex:fe,ff,01,00,44,ac,00,00,10,b1,\
  02,00,04,00,20,00,16,00,20,00,04,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
  aa,00,38,9b,71
"{F19F064D-082C-4E27-BC73-6882A1BB8E4C},0"=hex:fe,ff,01,00,44,ac,00,00,10,b1,\
  02,00,04,00,20,00,16,00,20,00,04,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
--
#time=1d5f73a4b70c2be
"DeviceState"=dword:00000001

[Software\\Microsoft\\Windows\\CurrentVersion\\MMDevices\\Audio\\Capture\\{3DDB3262-1A9F-4279-A79D-7E1742D69ED2}\\Properties] 1583877781
#time=1d5f727aa5675de
"{026E516E-B814-414B-83CD-856D6FEF4822},2"="In: Rocksmith USB Guitar Adapter - USB Audio"
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},0"=dword:00000004
"{233164C8-1B2C-4C7D-BC68-B671687A2567},1"="{3DDB3262-1A9F-4279-A79D-7E1742D69ED2}"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},14"="In: Rocksmith USB Guitar Adapter - USB Audio"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},2"="In: Rocksmith USB Guitar Adapter - USB Audio"
"{B3F8FA53-0004-438E-9003-51A46E139BFC},2"="{1}.USB\\VID_12BA&PID_00FF\\0&42D69ED2"
"{E4870E26-3CC5-4CD2-BA46-CA0A9A70ED04},3"=hex:fe,ff,02,00,80,bb,00,00,00,dc,\
  05,00,08,00,20,00,16,00,20,00,03,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
  aa,00,38,9b,71
"{F19F064D-082C-4E27-BC73-6882A1BB8E4C},0"=hex:fe,ff,02,00,80,bb,00,00,00,dc,\
--
#time=1d5f7264ac7137c
"DeviceState"=dword:00000004

[Software\\Microsoft\\Windows\\CurrentVersion\\MMDevices\\Audio\\Render\\{FD47D9CC-4218-4135-9CE2-0C195C87405B}\\Properties] 1583876540
#time=1d5f724c70e5848
"{026E516E-B814-414B-83CD-856D6FEF4822},2"="Pulseaudio"
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},0"=dword:00000001
"{1DA5D803-D492-4EDD-8C23-E0C0FFEE7F0E},3"=dword:00000003
"{233164C8-1B2C-4C7D-BC68-B671687A2567},1"="{FD47D9CC-4218-4135-9CE2-0C195C87405B}"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},14"="Pulseaudio"
"{A45C254E-DF1C-4EFD-8020-67D146A850E0},2"="Pulseaudio"
"{E4870E26-3CC5-4CD2-BA46-CA0A9A70ED04},3"=hex:fe,ff,02,00,44,ac,00,00,20,62,\
  05,00,08,00,20,00,16,00,20,00,03,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
  aa,00,38,9b,71
"{F19F064D-082C-4E27-BC73-6882A1BB8E4C},0"=hex:fe,ff,02,00,44,ac,00,00,20,62,\
  05,00,08,00,20,00,16,00,20,00,03,00,00,00,03,00,00,00,00,00,10,00,80,00,00,\
--
"SymbolicLinkValue"=hex(6):5c,00,52,00,45,00,47,00,49,00,53,00,54,00,52,00,59,\
  00,5c,00,4d,00,41,00,43,00,48,00,49,00,4e,00,45,00,5c,00,53,00,6f,00,66,00,\
  74,00,77,00,61,00,72,00,65,00,5c,00,57,00,69,00,6e,00,65,00,5c,00,50,00,6f,\
  00,72,00,74,00,73,00

[Software\\Wow6432Node\\Wow6432Node\\Ubisoft\\Rocksmith2014] 1583876436
#time=1d5f72489298714
"ExecutableName"="Rocksmith2014.exe"
"installdir"="Z:\\home\\svanbroekhoven\\steam_lib\\steamapps\\common\\Rocksmith2014"

[System\\CurrentControlSet\\Control\\Class\\{4d36e967-e325-11ce-bfc1-08002be10318}] 1554732615
#time=1d4ee14ca089e24
@="Disk drives"
"Class"="DiskDrive"

NS International move to Aws

Sat, 07 Mar 2020 16:09:42 +0100

The term lift and shift was thrown out the window and we’ve build a fully automated instructure build pipeline to facilitate the apps. From old jboss to modern docker based apps.

NS International, formerly NS Hispeed, is the rail operator in the Netherlands that operates international intercity and high-speed train connections with Belgium, France, Germany and Switzerland. NS International is part of the Dutch Railways. It currently operates both direct and indirect services. The largest online service they provide is ticket sales for international train journeys. From my role as an architect / sr. cloud engineer I am responsible for the design and delivery of the Cloud infrastructure platform based on AWS, in collaboration with the rest of our SRE team, which serves as an underlay for the digital ticket office. This role has been very diverse and has changed a lot over the years, when we started the transition to AWS, there was no cloud native plan to make the move possible, but only a list of two applications that eventually had to be moved to the aws platform. We were able to place these 2 applications (tomcat web frontend, jboss api backend) on AWS by creating an automatic AMI building process that uses ansible for the base configutation and prepares them for retrieving the ear / war application when starting up from S3. The platform has changed considerably twice, first with docker / ecs on an auto scaling ec2 cluster (I already designed this in 2016 and AWS now offers almost the same features in 2020) and finally ecs with fargate as a compute layer. The 16-year-old jboss app has also gone to ecs fargate with a lot of help from our developers, this was one of the jigsaw pieces to get this environment PCI DSS Compliant and we now officially have the stamp of PCI for our environment. The platform now has around 50 application stacks in production consisting of: spring boot, jboss, tomcat, s3 proxy, node and python.

Within this assignment I applied the following techniques: Cloudformation, Python, Ansible, bash, gitlab, azure devops, jira, docker, clair, server spec

Nederlands

NS International, voorheen NS Hispeed, is de spoor exploitant in Nederland die internationale intercity- en hogesnelheidstrein verbindingen met België, Frankrijk, Duitsland en Zwitserland exploiteert. NS International maakt deel uit van de Nederlandse Spoorwegen. Het exploiteert momenteel zowel snelle als niet-snelle diensten. De grootste online dienst die zij leveren is de kaartverkoop voor internationale treinreizen. Vanuit mijn rol als architect/sr. cloud engineer ben ik in samenwerking met het de rest van ons SRE team verantwoordelijk voor het design en de oplevering van het Cloud infra platform gebaseerd op AWS welke als onderlaag dienst voor het digitale loket voor de kaartverkoop. Deze rol is erg divers en veel veranderd door de jaren heen, toen we begonnen aan de transitie naar AWS was er nog geen cloud native plan om de verhuizing mogelijk te maken maar alleen een lijstje van twee applicaties die uiteindelijk verhuisd moesten worden naar het aws platform. We hebben deze 2 applicaties (tomcat web frontend, jboss api backend) op AWS kunnen neerzetten door een automatisch ami bouwprocess te maken die met ansible servers opbouwt en klaar zet voor het ophalen van de applicatie ear/war bij het opstarten uit S3. Het platform is inmiddels 2 keer behoorlijk veranderd, eerst met docker/ecs op een auto scaling ec2 cluster (ik heb dit 2016 al een keer ontworpen en AWS biedt bijna dezelfde functies nu net aan in 2020) en uiteindelijk ecs met fargate als compute laag. De jboss app van inmiddels 16 jaar oud is door ons team en hulp van developers ook naar ecs fargate gegaan, dit was een van de puzzel stukjes om deze omgeving PCI DSS Compliant te krijgen en inmiddels hebben we officieel de stempel van PCI voor onze omgeving. Het platform heeft inmiddels ongeveer 50 applicatie stacks in productie bestaande uit: springboot, jboss, tomcat, s3-proxy, node, python

Binnen deze opdracht heb ik de volgende technieken toegepast: Cloudformation, Python, Ansible, bash, gitlab, azure devops, jira, docker, clair, serverspec

AWS services: ECS, VPC, EC2, Fargate, RDS (aurora serverless/classic), Elasticache, ALBv2, Route53, Organization, S3, Cloudfront, IAM, SSM, Cloudwatch, ApplicationAutoScaling, SQS, EFS, CertificateManager, Lambda, ApiGateway, DynamoDB, ElasticBeanstalk, WAF

CPload

Thu, 05 Mar 2020 00:00:00 +0000

Testing with CPload

Stein van Broekhoven - Cloud People

Why even build this tool?

Other tools were:

not easy enough
bloated with too many options I didn’t need
not easy to hack on (by me)

Features I was looking for

use access logs for traffic generation
change fqdn to be able to hit specific endpoints
gradual increase of traffic
rewrite specific requests
custom pre script (build session)
respect cookies / sessions / tokens etc.
high throughput per cpu core #lean

Use cases

During platform migration
In your application testing suite
Rebuilding or splitting applications
Testing new infra components
Testing auto scaling

Setup

git clone https://gitlab.com/cloud-people/cpload.git
cd cpload
mkvenv || pip install --user -r requirements.txt

Filters

They’re defined in a python file called filters.py you need 2 steps to add a filter

Write a function that expands this basic one:

def uri_editor(http_pool, url):
    send_request(http_pool, url)

Add the function to the filter list

filters = {
    '/magic/': magic_edit,
    'api/v1/date': future,
    'web': to_upper,
    '': send_request
}

Filter example

Turn all urls containing api in to ALL CAPS

# ALL CAPS FOR ADDED DRAMA
def to_upper(http_pool, url):
    """URI TO UPPER"""
    send_request(http_pool, str(url).upper())

filters = {
    'api': to_upper,
    '': send_request
}

input='/api/v2/lalala/happy/url?data=test'
request='/API/V2/LALALA/HAPPY/URL?DATA=TEST'

What to feed this tool

Apache access logs
Nginx access logs
AWS ALB access logs
Hand crafted url list

A look at the options


optional arguments:
  -h, --help           show this help message and exit
  --urlfile URLFILE    the cleaned up http log
  --fromuri FROMURI    base uri/hostname as in the log file, this is used to edit it to the destination uri.
  --touri TOURI        the uri/hostname you want to send the traffic to
  --ratemin RATEMIN    minimum req/hour
  --ratemax RATEMAX    maximum req/hour
  --ramptime RAMPTIME  ramp up time in hours
  --duration DURATION  total test duration in hours
  --verbose            Output all request results
  --filters FILTERS    the uri filters and rewrites file

Now let’s fire it up

Load test with CPload

Tue, 03 Mar 2020 16:32:54 +0100

When building large complex infrastructures it becomes harder to validate if the performance you need can be provided by the system you’ve build. But customer demand for ever faster websites is growing by the day, how do you make sure you’ll be able to handle the next big sale or event on your web platform? I use a load generator that can replay access logs and has a way to slowly ramp up traffic to replicate a gradual or sudden inrush of traffic. Aside from the basics there are often uris you’d like to hit with a bit more precision than just replaying logs. That is where the CPload tool comes in, it has a python config file were you can run custom python code/http requests when specific uris pass through the log.

Some of the cases where you might want custom http requests in your load test

Steps in an order process where your end user has session data attached to their requests.
A date that only makes sense to be in the future could be automatically replaced with a date in the future.
If there is a certain set of backends you don’t want to hit in your test so you might want to exclude them or alter the requests that would normally hit this backend.

Use cases

During platform migration to validate stability before switchover.
In the application testing suite, to validate efficiency.
While rebuilding or splitting applications smaller components check if performance isn’t degraded.
When evaluating any new infra components, use it to compare performance/price to the old situation.

I use it mainly for testing the auto scaling configuration to verify if the extra infrastructures spins up fast enough to handle the increase in traffic. Also it is useful to have a super static traffic load and be able to see how many instances you need to handle that traffic, if this increases between releases have a look at where your code might be slowing things down and see if it is acceptable or should be tuned for more performance.

Why yet another tool

Other tools were not fixing my needs, were extremely expensive or had a way to broad feature set that I didn’t need. I needed something that runs everywhere with not to much hassle and had the following requirements:

use access logs for traffic generation
change the fqdn to be able to hit a specific endpoint
gradual increase of the traffic
rewrite specific requests
custom pre script (build session)
respect cookies / sessions / tokens etc.
high throughput per cpu core #lean

Examples

Let’s take a look at some examples, first off a simple http 200 responder written in python.

Clone the git repo

$ cd ~/git
$ git clone https://gitlab.com/cloud-people/cpload.git
$ cd cpload
$ tree
.
├── cpload.py
├── Dockerfile
├── examples
├── filters.py
├── LICENSE
├── logs.txt
├── README.md
└── requirements.txt

1 directory, 7 files
$

Get a http 200 responder container running

$ docker run -d -p 8080:8080 docker.io/aapjeisbaas/hello-container
9ed24540565844f6fc2de96f07c886caaf4cc358fd730936f718fbde56bc849e
$ curl 127.0.0.1:8080/kjkjkjkj
Your path is: /kjkjkjkj
$

Install requirements and run our first test

$ pip install -r requirements
...
...
$ python cpload.py --verbose --touri "http://127.0.0.1:8080" --ratemin 360 --ratemax 7200 --ramptime 0.1 --duration 0.2
Load test running at: 360 req/hour
200 http://127.0.0.1:8080/
200 http://127.0.0.1:8080/web/location/
200 http://127.0.0.1:8080/api/v1/location/?test=true
200 http://127.0.0.1:8080/web/location/nested/pages
200 http://127.0.0.1:8080/api/v1/date-sensitive/20200213/test
Load test running at: 873 req/hour
200 http://127.0.0.1:8080/
200 http://127.0.0.1:8080/api/v1/add/controlled/randomness/magic/test
200 http://127.0.0.1:8080/
200 http://127.0.0.1:8080/
200 http://127.0.0.1:8080/web/location/
200 http://127.0.0.1:8080/api/v1/location/?test=true
^C
Stopping load test at: 1272 req/hour

 $

Now let’s add the example url filters to edit some of the requests flowing through here.

$ python cpload.py --verbose --touri "http://127.0.0.1:8080" --ratemin 360 --ratemax 7200 --ramptime 0.1 --duration 0.2 --filters filters.py
loaded filters:

 function      help text                   uri filter
------------  --------------------------  ------------
magic_edit    Select only magic products  /magic/
future        Always use future dates     api/v1/date
to_upper      URI TO UPPER                web
send_request  Send the get request 

Load test running at: 360 req/hour
200 http://127.0.0.1:8080/
Load test running at: 379 req/hour
200 HTTP://127.0.0.1:8080/WEB/LOCATION/
200 http://127.0.0.1:8080/api/v1/location/?test=true
200 HTTP://127.0.0.1:8080/WEB/LOCATION/NESTED/PAGES
200 http://127.0.0.1:8080/api/v1/date-sensitive/20200317/test
Load test running at: 873 req/hour
200 http://127.0.0.1:8080/
200 http://127.0.0.1:8080/api/v1/add/controlled/randomness/magic1/test
200 http://127.0.0.1:8080/
^C
Stopping load test at: 1082 req/hour 

 $

the true power is in the filters

As you can see in the output different requests were sent to the target but both were using the same source logs.txt file. The url edits are done in the filters.py file, have a look at some of the examples in there:

print("loaded filters:")
# this file is included in the main cpload.py to add special sauce to your log replay.

# Here's the base function, what you do inside it is up to you.
#
# def uri_editor(http_pool, url):
#     send_request(http_pool, url)
#

# Here we select a random product from the magic list instead of using the one from the log.
# This might be helpfull when you have a small subset of the products in you tst / acc / mock environment
magicList = ['magic1', 'product2', 'destination3', 'package4']
def magic_edit(http_pool, url):
    """Select only magic products"""
    magic = random.choice(magicList)
    url = str(args.touri) + "/api/v1/add/controlled/randomness/" + magic + "/test"
    send_request(http_pool, url)

# In this example we replace a uri that is date sensitive with an url that always has a date in the future.
today = datetime.date.today()
def future(http_pool, url):
    """Always use future dates"""
    days = datetime.timedelta(days=random.randint(3, 20))
    date = today + days
    url = str(args.touri) + "/api/v1/date-sensitive/" + date.strftime("%Y%m%d") + "/test"
    send_request(http_pool, url)

# ALL CAPS FOR ADDED DRAMA
def to_upper(http_pool, url):
    """URI TO UPPER"""
    send_request(http_pool, str(url).upper())

# This is how the uri gets mapped to the correct editor function.
# It's selected based on if a string is string in uri, the first match will be used.
filters = {
    '/magic/': magic_edit,
    'api/v1/date': future,
    'web': to_upper,
    '': send_request
}

# draw a pretty overview of the loaded filters
from tabulate import tabulate
table = [["function", "help text", "uri filter"]]
for filter in filters:
    table.append([filters[filter].__name__, filters[filter].__doc__, filter])
print("\n", tabulate(table,headers="firstrow"), "\n")

Replay your own logs

To be able to replay your logs you need to clean them and make them look like the bellow example:

base.uri/
base.uri/web/location/
base.uri/api/v1/location/?test=true
base.uri/web/location/nested/pages
base.uri/api/v1/date-sensitive/20200213/test
base.uri/
base.uri/api/v1/add/controlled/randomness/magic/test
base.uri/

One liners to generate this from access logs.

nginx

10.0.2.2 - - [03/Mar/2020:14:28:05 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
2020/03/03 14:28:10 [error] 2#2: *1 open() "/usr/share/nginx/html/yolo" failed (2: No such file or directory), client: 10.0.2.2, server: localhost, request: "GET /yolo HTTP/1.1", host: "127.0.0.1:8080"
10.0.2.2 - - [03/Mar/2020:14:28:10 +0000] "GET /yolo HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
2020/03/03 14:28:13 [error] 2#2: *1 open() "/usr/share/nginx/html/yolo/test" failed (2: No such file or directory), client: 10.0.2.2, server: localhost, request: "GET /yolo/test HTTP/1.1", host: "127.0.0.1:8080"
10.0.2.2 - - [03/Mar/2020:14:28:13 +0000] "GET /yolo/test HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
2020/03/03 14:28:21 [error] 2#2: *1 open() "/usr/share/nginx/html/yolo/cloud-people" failed (2: No such file or directory), client: 10.0.2.2, server: localhost, request: "GET /yolo/cloud-people HTTP/1.1", host: "127.0.0.1:8080"
10.0.2.2 - - [03/Mar/2020:14:28:21 +0000] "GET /yolo/cloud-people HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
2020/03/03 14:28:26 [error] 2#2: *1 "/usr/share/nginx/html/yolo/cloud/index.html" is not found (2: No such file or directory), client: 10.0.2.2, server: localhost, request: "GET /yolo/cloud/ HTTP/1.1", host: "127.0.0.1:8080"
10.0.2.2 - - [03/Mar/2020:14:28:26 +0000] "GET /yolo/cloud/ HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
2020/03/03 14:28:31 [error] 2#2: *1 open() "/usr/share/nginx/html/static" failed (2: No such file or directory), client: 10.0.2.2, server: localhost, request: "GET /static HTTP/1.1", host: "127.0.0.1:8080"
10.0.2.2 - - [03/Mar/2020:14:28:31 +0000] "GET /static HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
2020/03/03 14:28:35 [error] 2#2: *1 open() "/usr/share/nginx/html/lalalala" failed (2: No such file or directory), client: 10.0.2.2, server: localhost, request: "GET /lalalala HTTP/1.1", host: "127.0.0.1:8080"
10.0.2.2 - - [03/Mar/2020:14:28:35 +0000] "GET /lalalala HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
2020/03/03 14:28:41 [error] 2#2: *1 open() "/usr/share/nginx/html/robots.txt" failed (2: No such file or directory), client: 10.0.2.2, server: localhost, request: "GET /robots.txt HTTP/1.1", host: "127.0.0.1:8080"
10.0.2.2 - - [03/Mar/2020:14:28:41 +0000] "GET /robots.txt HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.0.2.2 - - [03/Mar/2020:14:28:54 +0000] "GET /?test=true HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.0.2.2 - - [03/Mar/2020:14:28:58 +0000] "GET /?test=fals HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.0.2.2 - - [03/Mar/2020:14:29:00 +0000] "GET /?test=false HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.0.2.2 - - [03/Mar/2020:14:29:08 +0000] "GET /?search=a HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.0.2.2 - - [03/Mar/2020:14:29:10 +0000] "GET /?search=ab HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.0.2.2 - - [03/Mar/2020:14:29:12 +0000] "GET /?search=abc HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.0.2.2 - - [03/Mar/2020:14:29:13 +0000] "GET /?search=abcd HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.0.2.2 - - [03/Mar/2020:14:29:15 +0000] "GET /?search=abcde HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.0.2.2 - - [03/Mar/2020:14:29:18 +0000] "GET /?search=abcdef HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
2020/03/03 14:29:28 [error] 2#2: *1 open() "/usr/share/nginx/html/result/76353220" failed (2: No such file or directory), client: 10.0.2.2, server: localhost, request: "GET /result/76353220 HTTP/1.1", host: "127.0.0.1:8080"
10.0.2.2 - - [03/Mar/2020:14:29:28 +0000] "GET /result/76353220 HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"

$ docker logs 39c50e2d31ee | grep -ve '\[error\]' | grep 'GET\|POST' | awk '{print "base.uri" $7}'
base.uri/
base.uri/yolo
base.uri/yolo/test
base.uri/yolo/cloud-people
base.uri/yolo/cloud/
base.uri/static
base.uri/lalalala
base.uri/robots.txt
base.uri/?test=true
base.uri/?test=fals
base.uri/?test=false
base.uri/?search=a
base.uri/?search=ab
base.uri/?search=abc
base.uri/?search=abcd
base.uri/?search=abcde
base.uri/?search=abcdef
base.uri/result/76353220
$ docker logs 39c50e2d31ee | grep -ve '\[error\]' | grep 'GET\|POST' | awk '{print "base.uri" $7}' > nginx-logs.txt

Now run the test with the new log file:

$ python cpload.py --verbose --touri "http://127.0.0.1:8080" --ratemin 360 --ratemax 7200 --ramptime 0.1 --duration 0.2 --filters filters.py --urlfile nginx-logs.txt
loaded filters:

 function      help text                   uri filter
------------  --------------------------  ------------
magic_edit    Select only magic products  /magic/
future        Always use future dates     api/v1/date
to_upper      URI TO UPPER                web
send_request  Send the get request 

Load test running at: 360 req/hour
200 http://127.0.0.1:8080/
404 http://127.0.0.1:8080/yolo
404 http://127.0.0.1:8080/yolo/test
404 http://127.0.0.1:8080/yolo/cloud-people
404 http://127.0.0.1:8080/yolo/cloud/
Load test running at: 873 req/hour
404 http://127.0.0.1:8080/static
404 http://127.0.0.1:8080/lalalala
404 http://127.0.0.1:8080/robots.txt
Load test running at: 1082 req/hour
200 http://127.0.0.1:8080/?test=true
200 http://127.0.0.1:8080/?test=fals
200 http://127.0.0.1:8080/?test=false
200 http://127.0.0.1:8080/?search=a
200 http://127.0.0.1:8080/?search=ab
200 http://127.0.0.1:8080/?search=abc
200 http://127.0.0.1:8080/?search=abcd
Load test running at: 1462 req/hour
200 http://127.0.0.1:8080/?search=abcde
200 http://127.0.0.1:8080/?search=abcdef
404 http://127.0.0.1:8080/result/76353220
...
...
...

httpd / apache

Raw log:

AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.0.2.100. Set the 'ServerName' directive globally to suppress this message
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.0.2.100. Set the 'ServerName' directive globally to suppress this message
[Tue Mar 03 14:44:26.181291 2020] [mpm_event:notice] [pid 1:tid 140004244472960] AH00489: Apache/2.4.41 (Unix) configured -- resuming normal operations
[Tue Mar 03 14:44:26.183779 2020] [core:notice] [pid 1:tid 140004244472960] AH00094: Command line: 'httpd -D FOREGROUND'
10.0.2.2 - - [03/Mar/2020:14:44:46 +0000] "GET / HTTP/1.1" 200 45
10.0.2.2 - - [03/Mar/2020:14:44:55 +0000] "GET /yolo HTTP/1.1" 404 196
10.0.2.2 - - [03/Mar/2020:14:45:02 +0000] "GET /yolo/test HTTP/1.1" 404 196
10.0.2.2 - - [03/Mar/2020:14:45:07 +0000] "GET /yolo/cloud-people HTTP/1.1" 404 196
10.0.2.2 - - [03/Mar/2020:14:45:12 +0000] "GET /yolo/cloud/ HTTP/1.1" 404 196
10.0.2.2 - - [03/Mar/2020:14:45:16 +0000] "GET /static HTTP/1.1" 404 196
10.0.2.2 - - [03/Mar/2020:14:45:20 +0000] "GET /lalalala HTTP/1.1" 404 196
10.0.2.2 - - [03/Mar/2020:14:45:23 +0000] "GET /robots.txt HTTP/1.1" 404 196
10.0.2.2 - - [03/Mar/2020:14:45:27 +0000] "GET /?test=true HTTP/1.1" 200 45
10.0.2.2 - - [03/Mar/2020:14:45:30 +0000] "GET /?test=fals HTTP/1.1" 200 45
...
...
...

transform it

docker logs d5b6b17ba444 | grep 'GET\|POST' | awk '{print "base.uri" $7}'
base.uri/
base.uri/yolo
base.uri/yolo/test
base.uri/yolo/cloud-people
base.uri/yolo/cloud/
base.uri/static
base.uri/lalalala
base.uri/robots.txt
base.uri/?test=true
base.uri/?test=fals
base.uri/?test=false
base.uri/?search=a
base.uri/?search=ab
base.uri/?search=abc
base.uri/?search=abcd
base.uri/?search=abcde
base.uri/?search=abcdef
base.uri/result/76353220
base.uri/
base.uri/yolo
base.uri/yolo/test

AWS Alb

Here is an example to pull in aws alb logs from yesterday from your s3 logs bucket.

$ aws s3 --profile prd sync s3://your-logs-buckut/AWSLogs/youraccountnumber/elasticloadbalancing/yr-region-1/$(date --date yesterday "+%Y/%m/%d") . ; gzip -d *
...
...
...
$ # export all GET requests to your logs file
$ cat * | grep 'GET' |  awk '{print $14}' | sed 's/.*:443/base.uri/g' > alb-logs.txt

Cross build your docker images

Wed, 05 Feb 2020 17:01:16 +0100

Now that ARM is getting bigger and AWS offering compute on ARM for bargain prices I was wondering if it would be feasible to prepare most of my projects for a platform shift to ARM. The tool I like to use is buildx at the moment it is still experimental but I’m having great success with it so far. Docker Buildx is a CLI plugin that extends the docker command with the full support of the features provided by Moby BuildKit builder toolkit. It provides the same user experience as docker build with many new features like creating scoped builder instances and building against multiple nodes concurrently. And the most important thing to us it’s designed to work well with building for multiple platforms and not only for the architecture and operating system that the user invoking the build happens to run.

You can build multi-platform images using three different strategies that are supported by Buildx and Dockerfiles:

Using the QEMU emulation support in the kernel
Building on multiple native nodes using the same builder instance
Using a stage in Dockerfile to cross-compile to different architectures

We’ll focus on the first option, cross building with emulation.

docker buildx build --platform linux/amd64,linux/arm64 .

Setup

This differs from platform to platform but one thing we all have in common is pipelines, so I’ve constructed a basic buildx setup for GitLab CI/CD. If you want to crossbuild from your local machine have a look at this install guide.

Pipeline

This is a generic pipeline that takes in three variables that are pretty self explanatory:

DOCKERHUB_USER : dockerhub username
DOCKERHUB_PASS : dockerhub access token
DOCKERHUB_REPOSITORY full repository name like: aapjeisbaas/builder

build:
  image: docker:latest
  services:
  - docker:dind
  stage: build
  script:
    # install depends
    - apk add curl jq

    # enable experimental buildx features
    - export DOCKER_BUILDKIT=1
    - export DOCKER_CLI_EXPERIMENTAL=enabled

    # Download latest buildx bin from github
    - mkdir -p ~/.docker/cli-plugins/
    - BUILDX_LATEST_BIN_URI=$(curl -s -L https://github.com/docker/buildx/releases/latest | grep 'linux-amd64' | grep 'href' | sed 's/.*href="/https:\/\/github.com/g; s/amd64".*/amd64/g')
    - curl -s -L ${BUILDX_LATEST_BIN_URI} -o ~/.docker/cli-plugins/docker-buildx
    - chmod a+x ~/.docker/cli-plugins/docker-buildx

    # Get and run the latest docker/binfmt tag to use its qemu parts
    - BINFMT_IMAGE_TAG=$(curl -s https://registry.hub.docker.com/v2/repositories/docker/binfmt/tags | jq '.results | sort_by(.last_updated)[-1].name' -r)
    - docker run --rm --privileged docker/binfmt:${BINFMT_IMAGE_TAG}

    # create the multibuilder
    - docker buildx create --name multibuilder
    - docker buildx use multibuilder

    # login to a registry
    - docker login -u ${DOCKERHUB_USER} -p ${DOCKERHUB_PASS}

    # build the containers and push them to the registry then display the images
    - docker buildx build --platform linux/amd64,linux/arm64,linux/s390x,linux/386,linux/arm/v7,linux/arm/v6 -t ${DOCKERHUB_REPOSITORY}:latest . --push
    - docker buildx imagetools inspect ${DOCKERHUB_REPOSITORY}:latest

From here on out you can experiment with what platforms you can directly port to. Make sure you have a base image that supports multiarch like alpine or one of these base images.

Building for Raspberry pi

Make sure to include linux/arm/v7 for Raspberry Pi 2 and 3, older versions and pi zero use linux/arm/v6.

The Raspberry pi 4 will be able to use linux/arm/v7 and in the future if 64 bit arm becomes available might support linux/arm64

More resources

Display AWS Cloudformation logs in pipelines

Fri, 10 Jan 2020 00:16:46 +0100

Using cloudformation is great, it gives you insane super powers but there are some small things that annoy me when using it in pipelines. My biggest issue is that it is not displaying the steps it takes in kind of real time’ish to stdout, so that’s where the following snippet will help.

The code

What the output looks like

[
    {
        "StackId": "arn:aws:cloudformation:eu-west-1:368341477005:stack/tech-blog/2b819be0-28fe-11ea-8ab3-06e58f87e324",
        "EventId": "5816a850-3333-11ea-83e6-02bbe9b74d60",
        "StackName": "tech-blog",
        "LogicalResourceId": "tech-blog",
        "PhysicalResourceId": "arn:aws:cloudformation:eu-west-1:368341477005:stack/tech-blog/2b819be0-28fe-11ea-8ab3-06e58f87e324",
        "ResourceType": "AWS::CloudFormation::Stack",
        "Timestamp": "2020-01-09T22:57:00.232Z",
        "ResourceStatus": "UPDATE_IN_PROGRESS",
        "ResourceStatusReason": "User Initiated"
    }
]


[
    {
        "StackId": "arn:aws:cloudformation:eu-west-1:368341477005:stack/tech-blog/2b819be0-28fe-11ea-8ab3-06e58f87e324",
        "EventId": "5af0ca60-3333-11ea-94ed-0655dfaca74c",
        "StackName": "tech-blog",
        "LogicalResourceId": "tech-blog",
        "PhysicalResourceId": "arn:aws:cloudformation:eu-west-1:368341477005:stack/tech-blog/2b819be0-28fe-11ea-8ab3-06e58f87e324",
        "ResourceType": "AWS::CloudFormation::Stack",
        "Timestamp": "2020-01-09T22:57:05.016Z",
        "ResourceStatus": "UPDATE_COMPLETE_CLEANUP_IN_PROGRESS"
    }
]


[
    {
        "StackId": "arn:aws:cloudformation:eu-west-1:368341477005:stack/tech-blog/2b819be0-28fe-11ea-8ab3-06e58f87e324",
        "EventId": "5b481220-3333-11ea-ac13-02d3b89dc706",
        "StackName": "tech-blog",
        "LogicalResourceId": "tech-blog",
        "PhysicalResourceId": "arn:aws:cloudformation:eu-west-1:368341477005:stack/tech-blog/2b819be0-28fe-11ea-8ab3-06e58f87e324",
        "ResourceType": "AWS::CloudFormation::Stack",
        "Timestamp": "2020-01-09T22:57:05.588Z",
        "ResourceStatus": "UPDATE_COMPLETE"
    }
]



Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - tech-blog
section_end:1578610649:build_script

If you want it all and you want it now

This is the easiest way to use this snippet, I will try to keep functionality stable for this snippet but there are no guarantees that this wil work till the end of time.

#!/bin/bash

# pull in cfn_wrapper
source <(curl -s https://gitlab.com/snippets/1928583/raw)

# deploy
cfn_wrapper aws cloudformation deploy --template-file "cloudformation.yml" \
  --stack-name $STACK --capabilities='CAPABILITY_IAM' --no-fail-on-empty-changeset \
  --parameter-overrides \
  CfnParam="$CFN_PARAM" \
  CfnVar="$SOME_VAR"

Create your own Wifi QR

Sun, 22 Dec 2019 16:59:39 +0100

If you’re tired of handing out wifi credentials on paper maybe this trick will help you out.

Example with qrencode

This tool creates a qr in your terminal if that’s something you might need:

   echo 'WIFI:S:The Promised Lan;T:WPA;P:Thisisthekey!;;' | qrencode -t ansiutf8
█████████████████████████████████████
█████████████████████████████████████
████ ▄▄▄▄▄ █▄▀ ▀ ▄ ▀█ ▀███ ▄▄▄▄▄ ████
████ █   █ █   █▀▀▄▀▀  █ █ █   █ ████
████ █▄▄▄█ █▄█▀ ▄▄█ █▀█▄▄█ █▄▄▄█ ████
████▄▄▄▄▄▄▄█▄█ █ █ █▄▀▄█ █▄▄▄▄▄▄▄████
████▄   ▀▀▄█ █ ▄▀█▀▄█▀█ ▄▄▀█  ▄█ ████
████ ▀█ ▄▄▄▀   ▄█▀█▄█▄ █  ██▀▀█ ▄████
████▄▀██ ▀▄▀ ▄▄▀▄▄█ ▄▄▄▀▀   █ ▄▄ ████
█████▀ ▄ █▄▄▄ ▀█▄▀█▀  █▄█  ██  ▄▄████
████▄ ▄▀▄ ▄▀ ▄ ▄█  ▀ ▀█▀█▀▄▄▄▀ ▄█████
████▄█▀▀▀▀▄▀▀█  ▀▀▄ █▀▄▀▀██▀▄█  ▀████
█████▄█▄▄█▄▄ ▀█▄ █▀▄▄▄▀  ▄▄▄ ▀███████
████ ▄▄▄▄▄ ██▄▄█ ▄▄█▄ █  █▄█ ▀▄▄▀████
████ █   █ █▀▀▀ █ ██▄█ ▄▄▄ ▄▄  ██████
████ █▄▄▄█ █  █▄▀ ▄▀▄▀▄ ▀ ▀█▄▄█▀ ████
████▄▄▄▄▄▄▄█▄█▄▄▄█▄▄█████▄█▄█▄███████
█████████████████████████████████████
█████████████████████████████████████

qrencode

Make it good looking with a python tool

Use this wifi logo for the best results:

myqr 'WIFI:S:The Promised Lan;T:WPA;P:Thisisthekey!;;' -p wifi-logo.png -n wifi-qr.jpg

MyQR result

Update all docker images

Sun, 22 Dec 2019 16:52:28 +0100

When you’re running images with the tag latest on you local machine they might need some updates from time to time. With this one liner you pull the newest version in of all the images you already have locally.

Update all images

for image in $(sudo docker image ls | awk '{print $1}' | grep -ve "none\|REPOSITORY" | sort | uniq); do sudo docker pull $image ;done

example

This was on my media management machine:

[svanbroekhoven@ibm729 ~]$ sudo docker image ls | awk '{print $1}' | grep -ve "none\|REPOSITORY" | sort | uniq
linuxserver/nzbget
linuxserver/plex
linuxserver/radarr
linuxserver/sonarr
linuxserver/tautulli
nvidia/cuda
ortho4xp
swarm
ubuntu
[svanbroekhoven@ibm729 ~]$ for image in $(sudo docker image ls | awk '{print $1}' | grep -ve "none\|REPOSITORY" | sort | uniq); do sudo docker pull $image ;done
Using default tag: latest
latest: Pulling from linuxserver/nzbget
Digest: sha256:4c433da392324f885dcb63b7a2aecaa72125a172ffb81761cf34dd3462dd7c65
Status: Image is up to date for linuxserver/nzbget:latest
docker.io/linuxserver/nzbget:latest
Using default tag: latest
latest: Pulling from linuxserver/plex
Digest: sha256:158865b029f109dc9f9460de792d78f1df056188ebf6a720a0bdc1ccde768f2a
Status: Image is up to date for linuxserver/plex:latest
docker.io/linuxserver/plex:latest
Using default tag: latest
latest: Pulling from linuxserver/radarr
Digest: sha256:8dbe3385f3bcd8d16558f6f9e31224f1d3ee994debb90454a7b827c745ff3c6d
Status: Image is up to date for linuxserver/radarr:latest
docker.io/linuxserver/radarr:latest
Using default tag: latest
latest: Pulling from linuxserver/sonarr
Digest: sha256:3c57fca1f14943219b2a4573464ab462ac60b8d8fd0396097b61ef7fc6366970
Status: Image is up to date for linuxserver/sonarr:latest
docker.io/linuxserver/sonarr:latest
Using default tag: latest
latest: Pulling from linuxserver/tautulli
Digest: sha256:9e3201491b9e6dce3ab476145a24371ce420e44ad042614ba46a69ffe3168672
Status: Image is up to date for linuxserver/tautulli:latest
docker.io/linuxserver/tautulli:latest
Using default tag: latest
latest: Pulling from nvidia/cuda
Digest: sha256:31e2a1ca7b0e1f678fb1dd0c985b4223273f7c0f3dbde60053b371e2a1aee2cd
Status: Image is up to date for nvidia/cuda:latest
docker.io/nvidia/cuda:latest
Using default tag: latest
Error response from daemon: pull access denied for ortho4xp, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Using default tag: latest
latest: Pulling from library/swarm
Digest: sha256:b866583a3b8791bcd705b7bc0fd94c66b695a1a2dbaeb5f59ed29940e5015dc8
Status: Image is up to date for swarm:latest
docker.io/library/swarm:latest
Using default tag: latest
latest: Pulling from library/ubuntu
Digest: sha256:250cc6f3f3ffc5cdaa9d8f4946ac79821aafb4d3afc93928f0de9336eba21aa4
Status: Image is up to date for ubuntu:latest
docker.io/library/ubuntu:latest
[svanbroekhoven@ibm729 ~]$

Mass delete cloudformation stacks

Wed, 13 Nov 2019 11:32:37 +0100

The oneliner:

# This will delete all stacks in the account connected to yourprofile with the string "delete-me" in the stackname
for StackName in $(aws --profile yourprofile cloudformation list-stacks --query 'StackSummaries[?contains(StackName, `delete-me`) == `true`] .StackName' --output text); do echo $StackName ; aws --profile yourprofile cloudformation delete-stack --stack-name $StackName ; done

Cloudformation & Autoscaling

Tue, 01 Oct 2019 17:30:00 +0100

Cloudformation and auto scaling, from the concepts to the details

Wed, 11 Sep 2019 18:46:23 +0200

Before you start

Not every environment is ready for auto scaling, to be able to fully utilize these techniques your infra must comply with the following.

No local state on the disks of application servers.
Redeployable images: ec2 asg/spot fleet (ami) or in the case of ecs you need docker images. In case you want something to get started with, I would recommend using the linked below “hello world” docker image to get familiar with the way of working.

Hello container

Horizontal vs Vertical scaling

The basic definition which can be applied in almost all cases:

Vertical scaling: You increase the compute power of the node your code runs on. This type of scaling is pretty limited and should be avoided is possible.

Horizontal scaling: You increase the amount of nodes handling your traffic while keeping the per node size small. This increases your resilience as your traffic increases and if a node fails impact is lower, this is the type of scaling we will take a look at.

Why would you add automated scaling

The answer most people will tell you is; “To handle peak workloads.” but this is not the most important part of it. After thinking it over I came to these 2 key reasons why auto scaling is the way to go.

Predictability
Flexibility

Predictability example: When your application in it’s most minimal, predictable and stable form can handle a maximum of X req/s adding more nodes will theoretically give you NX req/s maximum throughput. Of course other bottlenecks will appear like database throughput, hold that thought I will go into detail about automatically scaling your database later in this article.

Predictability example: As your traffic increases you automatically add more vault tolerance, scaling out with small instances will decrease the impact of a node failure. Node failure is a given, it will happen you just don’t know when, if you don’t have automated scaling you might turn off the node because it is not functioning and put that load permanently on the rest of the cluster until the node is fixed.

Flexibility example: If you optimize your application you can easily change the machine size as you are already adding and subtracting machines regularly this is nothing out of the ordinary and could change many times a day. This makes size selecting based on trial and measure iterations shorter and leads to faster results.

Flexibility example: As your application and requirements changes the need may arise to deploy to a different part of the world to have lower latency for your end users or migrate to a new location for whatever reason, it is pretty easy to add a new auto scaling resource in a different location and reroute your traffic to the new location.

What to scale

You can scale many parts of your infrastructure, in this writeup I focus on the following:

Types of scaling triggers

Step based scaling

You define a set threshold if it is crossed scaling takes an action.

Target tracking

You define a number and a metric that should lower or raise by increasing or decreasing the amount of application servers and let auto scale follow your desired number This in the background sets 2 step based scaling rules 1 to scale out and another to scale in.

What to base the scaling on

CPU Usage
ALB traffic
Memory usage
Cloudwatch metrics

What does scaling look in a graph

No scaling

Perfect scaling

Scaling parameters

This scaling examples was based on the following calculation: B3=IF(A2 < 2*D2, 3, ROUNDUP(A2/D2) this roughly translates to the following:

Track 40 req/node with a minimum of 3 nodes

As auto scaling is reactive not predictive you first see the the traffic increase and in the next increment you see the instances respond. The example just calculates the desired amount based on the previous data point, in AWS you have quite a bit more control on the triggers. (if you want) Scaling steps are based on cloudwatch alarms, so if you can create a cloudwatch alarm for something you can use it to base your scaling on.

Period is the length of time to evaluate the metric or expression to create each individual data point for an alarm. It is expressed in seconds. If you choose one minute as the period, there is one data point every minute.
Evaluation Period is the number of the most recent periods, or data points, to evaluate when determining alarm state.
Datapoints to Alarm is the number of data points within the evaluation period that must be breaching to cause the alarm to go to the ALARM state. The breaching data points don’t have to be consecutive, they just must all be within the last number of data points equal to Evaluation Period.

How to write the scaling rules in Cloudformation

The most straight forward autoscaling can be found in EC2 Auto Scaling groups, lets first take a look at that and later on dive into more complex setups.

AWSTemplateFormatVersion: 2010-09-09
Parameters:
  AMI:
    Type: String
  Subnets:
    Type: CommaDelimitedList
  AZs:
    Type: CommaDelimitedList
  PolicyTargetValue:
    Type: String
Resources:
  # what the instances should look like
  myLaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      ImageId: !Ref AMI
      InstanceType: t3.micro

  # group the instances
  myASGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MaxSize: '2'
      AvailabilityZones: !Ref AZs
      VPCZoneIdentifier: !Ref Subnets
      MinSize: '1'
      LaunchConfigurationName: !Ref myLaunchConfig

  # Scale the group by this rule
  myCPUPolicy:
    Type: AWS::AutoScaling::ScalingPolicy
    Properties:
      AutoScalingGroupName: !Ref myASGroup
      PolicyType: TargetTrackingScaling
      TargetTrackingConfiguration:
        PredefinedMetricSpecification:
          PredefinedMetricType: ASGAverageCPUUtilization
        TargetValue: !Ref PolicyTargetValue

This is probably the most compact way to write an auto scaling application stack based on machine CPU load as a variable, it uses 3 parts:

Resource to scale
Grouping those resources
Scaling rules for the group

Unfortunately, this is not the case for all things we want to scale, the other type of scaling is based on Application Auto Scaling which is based on the following components:

Resource to scale
Grouping those resources
Scaling rules for the group

Lets go trough one of my most elaborate self scaling templates, piece by piece.

We start of with a ecs service, I left out the boilerplate resources like task definition and iam roles. For the full templates check out the links at the bottom of this article.

  Service:
    Type: AWS::ECS::Service
    Properties:
      TaskDefinition: !Ref TaskDefinition
      DesiredCount: !Ref DesiredCount
      HealthCheckGracePeriodSeconds: 600
      LoadBalancers:
      - TargetGroupArn: !Ref TargetGroup
        ContainerPort: !Ref ContainerPort
        ContainerName: !Ref ContainerName
      Cluster: !Ref ContainerCluster
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: DISABLED
          SecurityGroups:
          - !Ref ContainerSG
          Subnets: !Ref Subnets
    DependsOn: ListenerRule

This is the service that we want to scale, it is located in fargate so aside from soft limits of containers per account not much scaling restrictions. Sorry I’ll not go deeper into building ECS services as it is a bit outside of the scope of this article.

By default the ecs service resource has no scaling mechanism, so we create one with the AWS::ApplicationAutoScaling::ScalableTarget resource. This puts a virtual handle on a dimension of your resource which will be controllable by auto scaling.

  ServiceScalingTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties:
      MinCapacity: !Ref MinCount
      MaxCapacity: !Ref MaxCount
      ResourceId:
        # Example: service/MyECSCluster-AB12CDE3F4GH/MyECSService-AB12CDE3F4GH
        Fn::Join:
        - ''
        - - service/
          - Ref: ContainerCluster
          - "/"
          - Fn::GetAtt:
            - Service
            - Name
      RoleARN:
        Fn::GetAtt:
        - ApplicationAutoScalingRole
        - Arn
      ScalableDimension: ecs:service:DesiredCount
      # The all important scaling Dimension, in this case the amount of containers running in an ECS service.
      ServiceNamespace: ecs
      ScheduledActions:
        Fn::If:
        - NonProdCondition
        -
          - ScalableTargetAction:
              MinCapacity: 0
              MaxCapacity: 0
            Schedule: "cron(0 20 * * ? *)"
            ScheduledActionName: NightlyDown
          - ScalableTargetAction:
              MinCapacity: !Ref MinCount
              MaxCapacity: !Ref MaxCount
            Schedule: "cron(30 4 * * ? *)"
            ScheduledActionName: MorningUp
        - Ref: AWS::NoValue

This ScalableTarget gives us a few more options to scale our service, as you can see in the example above I use a cron like scaling action to turn off the service outside working hours if it isn’t a production deployment. Of course you can go all out with the cron function and scale your application that way, but that’s not really automatic but can help if you have no trust in your auto scaling or to give your auto scaling reasonable boundaries to operate in.

So Let’s break it down into it’s components and where they get there data from

  ServiceScalingTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties:
      MinCapacity: !Ref MinCount
      MaxCapacity: !Ref MaxCount
      ResourceId:
        Fn::Join:
        - ''
        - - service/
          - Ref: ContainerCluster
          - "/"
          - Fn::GetAtt:
            - Service
            - Name

Capacity: Pretty straight forward, these are the upper and lower boundaries I import them from a variable with !Ref
ResourceId This is in our case a “pointer” to the ecs service, but instead of a simple !Ref we need a RecourceId

Unfortunately we can’t just ask for this string, we need to construct it ourself so lets start with the end result and work our way backwards on how to construct that string.

service/container-cluster/container-service
  |     \                                /
  |      +--- Unique Identifier --------+
  |
  +-- resource type

      RoleARN:
        Fn::GetAtt:
        - ApplicationAutoScalingRole
        - Arn
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      ScheduledActions:
        Fn::If:
        - NonProdCondition
        -
          - ScalableTargetAction:
              MinCapacity: 0
              MaxCapacity: 0
            Schedule: "cron(0 20 * * ? *)"
            ScheduledActionName: NightlyDown
          - ScalableTargetAction:
              MinCapacity: !Ref MinCount
              MaxCapacity: !Ref MaxCount
            Schedule: "cron(30 4 * * ? *)"
            ScheduledActionName: MorningUp
        - Ref: AWS::NoValue

Now that we have the groundwork in place lets add some automation to scale based on target tracking

  ServiceScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyName: !Sub ${AWS::StackName}
      PolicyType: TargetTrackingScaling
      ScalingTargetId: !Ref ServiceScalingTarget
      TargetTrackingScalingPolicyConfiguration:
        TargetValue: !Ref AutoscalingCpuTarget
        ScaleInCooldown: 180
        ScaleOutCooldown: 30
        PredefinedMetricSpecification:
          PredefinedMetricType: ECSServiceAverageCPUUtilization

This is the most basic auto scaling for ecs services and is pretty effective, especially if you consider how easy it is to implement. Let’s deconstruct it

  ServiceScalingPolicyReqPerM:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyName: 'req_per_minute'
      PolicyType: TargetTrackingScaling
      ScalingTargetId: !Ref ServiceScalingTarget
      TargetTrackingScalingPolicyConfiguration:
        TargetValue: !Ref AutoscalingReqTarget
        ScaleInCooldown: 180
        ScaleOutCooldown: 30
        PredefinedMetricSpecification:
          PredefinedMetricType: ALBRequestCountPerTarget
          ResourceLabel: !Join [ '/', [!ImportValue GulbFullName, !GetAtt TargetGroup.TargetGroupFullName] ]

          # https://docs.aws.amazon.com/en_pv/AWSCloudFormation/latest/UserGuide/aws-properties-applicationautoscaling-scalingpolicy-predefinedmetricspecification.html
          # RecourceLabel:
          # app/<load-balancer-name>/<load-balancer-id>/targetgroup/<target-group-name>/<target-group-id>

          # arn:aws:elasticloadbalancing:eu-west-1:716268079250:loadbalancer/app/gulb/8a5d787993154763
          # app/gulb/8a5d787993154763
          # !ImportValue GulbFullName

          # arn:aws:elasticloadbalancing:eu-west-1:716268079250:targetgroup/iris-Targe-2ITUDAYHYT85/17d40dfaff1a60e6
          # targetgroup/iris-Targe-2ITUDAYHYT85/17d40dfaff1a60e6
          # !GetAtt TargetGroup.TargetGroupFullName

          # RecourceLabel:
          # app/gulb/8a5d787993154763/targetgroup/iris-Targe-2ITUDAYHYT85/17d40dfaff1a60e6
          # !Join [ '', [!ImportValue GulbFullName, !GetAtt TargetGroup.TargetGroupFullName] ]

Add target tracking auto scale to your aws ecs service

Mon, 04 Feb 2019 11:59:01 +0100

This cloudformation snippet will add auto scale to your existing cloudformation based ecs service.

To connect it to your service you may need to edit the folowing vars ServiceScalingTarget:

Service This is the name of the cloudformation block that defines your ecs service
Ref: ContainerCluster This should pull in the ecs cluster name I define this in a param: ContainerCluster

  ApplicationAutoScalingRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service: [application-autoscaling.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: /
      Policies:
      -
        PolicyName: ECSBlogScalingRole
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - ecs:UpdateService
            - ecs:DescribeServices
            - application-autoscaling:*
            - cloudwatch:DescribeAlarms
            - cloudwatch:GetMetricStatistics
            Resource: "*"

  ServiceScalingTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties:
      MaxCapacity: 20
      MinCapacity: 1
      ResourceId:
        Fn::Join:
        - ''
        - - service/
          - Ref: ContainerCluster
          - "/"
          - Fn::GetAtt:
            - Service
            - Name
      RoleARN:
        Fn::GetAtt:
        - ApplicationAutoScalingRole
        - Arn
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs

  ServiceScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyName: !Sub ${AWS::StackName}
      PolicyType: TargetTrackingScaling
      ScalingTargetId: !Ref ServiceScalingTarget
      TargetTrackingScalingPolicyConfiguration:
        TargetValue: 50.0
        ScaleInCooldown: 60
        ScaleOutCooldown: 60
        PredefinedMetricSpecification:
          PredefinedMetricType: ECSServiceAverageCPUUtilization

Hyper fast wordpress

Mon, 15 Oct 2018 22:56:54 +0200

How to speedup wordpress without without changing your website.

Not one of my regular deeply technical posts but not any less valueable.

You’re probably here because you have a Wordpress based website and are not too impressed with the performance. There are a few approaches to make your content Hyper Fast ;-)

use a cache plugin with minimal changes to the site
use a cache plugin to build a static version of the site
migrate content to a static site generator ( hugo / pelican / jekyll)
cache your site in a cdn and purge when you update content

Today we are looking into the last one, cache everything at a cdn and purge on updates. Let’s go trough the steps:

Register at cloudflare.com

This is where your content will be cached.

Setup DNS and CDN

After you’ve created your account you need to start using cloudflare’s dns servers. This if done carefully, can be done without any downtime or hickups.

Create the domain in your cloudflare account (this is in the sign up procedure if I recall correctly)
Copy all the DNS records from your existing DNS provider to cloudflare trough their excelent web interface. Except the NS records, we’ll do these last.
Make sure you have no orange clouds behind your records, we will enable their cdn later.
You can make some dns requests with dig to compare old and new dns servers: (you can find your cloudflare dns servers in the web pannel)

domain="aapjeisbaas.nl"
# find your current nameservers:
dig +short ns $domain

# find your cloudflare nameservers:
dig +short ns $domain @ns.cloudflare.com.

# compare some records
dig +short a $domain 
dig +short a $domain @ns.cloudflare.com.

dig +short a www.$domain 
dig +short a www.$domain @ns.cloudflare.com.

dig +short mx $domain 
dig +short mx $domain @ns.cloudflare.com.

dig +short txt $domain 
dig +short txt $domain @ns.cloudflare.com.

If the results from your current and cloudflare’s dns servers match you can carry on to the next step.

Start using cloudflare’s nameservers

So your old and new DNS setup match, now we need to replace the NS records at your registrar. Most of the time the place where you bought your domain has a way to change your nameservers. Change them to the records provided by cloudflare, you can double check with:

# find your cloudflare nameservers:
dig +short ns $domain @ns.cloudflare.com.

To check if you changed your Nameservers correctly at your registrar use the following commands, keep in mind that this may take a few hours to be reflected in the results:

# output should contain cloudflare nameservers
whois $domain

# output should look like:
# aida.ns.cloudflare.com. dns.cloudflare.com. 2029044598 10000 2400 604800 3600
# \_ Your 1st nameserver   \_ contact          \_ serial  \     \    \      \_ ttl
#                                                          \     \    \_ expire
#  Changes in your dns wil increase the serial              \     \_ retry
#                                                            \_ refresh
dig +short soa $domain
dig +short soa $domain @1.1.1.1
dig +short soa $domain @8.8.8.8
dig +short soa $domain @4.2.2.2


# output should only contain your cloudflare nameservers
dig +short ns $domain
dig +short ns $domain @1.1.1.1
dig +short ns $domain @8.8.8.8
dig +short ns $domain @4.2.2.2

Test it

Try to send some mail cross domains, recieve some mail cross domains. visit your website from multiple machines and devices.

Enable cloudflare

To use cloudflare’s CDN goodness simply click on the cloud behind a DNS record in the dns admin interface. When it is orange with the arrow trough it, it is active.

Install 2 plugins to get the most out of this setup

To cache your pages at cloudflare for as long as you want: https://wordpress.org/plugins/cache-control/ Increase s-maxage to 604800 to cache for 1 week, you can leave the other options default.

To clear the cache when you update your site: https://wordpress.org/plugins/cloudflare/ After install connect it with your cloudflare account to allow your site to clear the cache. When your api connection is set up click the optimize for wordpress button to tweak some settings.

Take it up a notch

To cache all the pages you serve create a page rule for *testdomain.com/* with setting Cache level: Cache Everything

If you have any questions feel free to leave them below, if you want personal advice on how to optimize your specific website contact me

Get ec2 arn for local ec2 machine with python

Sun, 16 Sep 2018 16:22:45 +0200

A super minimal urllib based arn constructor based on local metadata

import urllib.request

# local ec2 arn:
f = urllib.request.urlopen('http://169.254.169.254/latest/meta-data/placement/availability-zone')
az = f.read().decode('utf-8')
f = urllib.request.urlopen('http://169.254.169.254/latest/meta-data/instance-id')
instance = f.read().decode('utf-8')
arn = 'arn:aws:ec2:' + az + ':instance/' + instance
print(arn)

This returns arn as used by the EC2 Spot Instance Interruption Warning event

{
  "version": "0",
  "id": "03a51c99-95cb-c6e8-03a8-3ce3ac402d64",
  "detail-type": "EC2 Spot Instance Interruption Warning",
  "source": "aws.ec2",
  "account": "763603311319",
  "time": "2018-09-16T13:17:10Z",
  "region": "eu-west-1",
  "resources": [
    "arn:aws:ec2:eu-west-1b:instance/i-05493b58ebcba0c43"
  ],
  "detail": {
    "instance-id": "i-05493b58ebcba0c43",
    "instance-action": "terminate"
  }
}

aws cli search for latest ami

Fri, 24 Aug 2018 21:28:27 +0200

This simple oneliner gets you the latest ami in a given region for a specific search term.

# get the latest ecs-optimized amazon ami in eu-west-1
aws ec2 describe-images --region eu-west-1 --owners amazon --filters "Name=name,Values=*amazon-ecs-optimized" "Name=architecture,Values=x86_64" "Name=architecture,Values=x86_64" --query 'sort_by(Images, &CreationDate)[-1].ImageId'

So lets break it down to give you what you need to get your favorite ami.

# Get the full ami list for name search term "*amazon-ecs-optimized"
aws ec2 describe-images --region eu-west-1 --owners amazon --filters "Name=name,Values=*amazon-ecs-optimized" "Name=architecture,Values=x86_64"

# this is searching for just the name, you could of course replace or expand this with other atributes like:
aws ec2 describe-images --region eu-west-1 --owners amazon --filters "Name=name,Values=*amazon-ecs-optimized" "Name=architecture,Values=x86_64" "Name=block-device-mapping.volume-type,Values=gp2"

the full filter atributes list can be found in the aws docs

Now lets take a look at the structure of the returned json

aws ec2 describe-images --region eu-west-1 --owners amazon --filters "Name=name,Values=*amazon-ecs-optimized" "Name=architecture,Values=x86_64" | head

# example output:
{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2017-11-10T19:59:12.000Z",
            "ImageId": "ami-014ae578",
            "ImageLocation": "amazon/amzn-ami-2017.09.b-amazon-ecs-optimized",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "591542846629",

This returnes an Images array with your results (ami with all the juicy metadata)

I use the built in query tool to grok this data (the query is written in JMESPath), to get at the object that we want we take the following steps:

sort an array
which one: the Images array
what to sort that data by the value of CreationDate
return the last object
retrieve .ImageId from the returned object

return objects from the Images array sorted by value of date

--query 'sort_by(Images, &CreationDate)'

just get the -1 object (-1 last, 0 first)

--query 'sort_by(Images, &CreationDate)[-1]'

get the ami id from the returned object

--query 'sort_by(Images, &CreationDate)[-1].ImageId'

Testing these query’s can be done over here: JMESPath

Now put all this together and you get:

$ aws ec2 describe-images --owners amazon --filters "Name=name,Values=*amazon-ecs-optimized" "Name=architecture,Values=x86_64" --query 'sort_by(Images, &CreationDate)[-1].ImageId'
"ami-0af844a965e5738db"

Elastic beanstalk latest PHP SolutionStackName finder

aws --region us-east-2 elasticbeanstalk list-available-solution-stacks --query 'SolutionStackDetails[?contains(SolutionStackName, `PHP`) == `true`].[SolutionStackName][-1]'

Figuring this out took me way too long so thought I’d share it here

More examples I found in a blogpost of OpenSource Connections

Presentaties en trainingen

Thu, 28 Jun 2018 00:00:00 +0100

Allereerst wat of wie is Aapje is Baas?

Aapje is Baas is een eenmanszaak van Stein van Broekhoven.

De bedrijfsnaam is ontstaan door een succesvolle test die toen dit domein registreerde en het is blijven hangen. 😉

Heeft u een onderwerp waar uw organisatie in kan groeien en wilt u hier een presentatie over bij u op locatie? Ik help u hier graag bij en heb een breed kennis gebied waarover ik graag vertel en uitleg.

Onderwerpen

CI/CD
AWS Cloud
Microservice architecture
KISS IT
Infra as Code
Compliance in moderne infra
Release cycle versnellen
Site Reliability Engineering

Wilt u een afspraak maken om te kijken wat ik voor u kan beteken, neem gerust contact op.

Install vim without X11 on FreeBSD

Thu, 28 Sep 2017 15:09:32 +0200

I can work with vi, but after using vim for years I just can’t work without it. The only problem is, if you install vim with pkg install vim it installs X11 and related crap.

This should be done as root.

First update portsnap:

# get it
portsnap fetch update

# extract it
portsnap extract

root@bsd-test:~ # cd /usr/ports/editors/vim
root@bsd-test:/usr/ports/editors/vim # ll
total 27
-rw-r--r--  1 root  wheel  7022 Sep 23 23:06 Makefile
-rw-r--r--  1 root  wheel   175 Sep 23 23:06 distinfo
drwxr-xr-x  2 root  wheel     6 Sep 28 14:25 files/
-rw-r--r--  1 root  wheel   601 Jun 14  2015 pkg-descr
-rw-r--r--  1 root  wheel  8897 Jan  9  2017 pkg-plist
root@bsd-test:/usr/ports/editors/vim # make WITHOUT_X11=yes install clean

For the option questions I only changed the first one to console and not using any graphic stuff.

It complained it couldn’t find perl5.24.3, so I installed it:

root@bsd-test:/usr/ports/editors/vim # cd /usr/ports/lang/perl5.24
root@bsd-test:/usr/ports/lang/perl5.24 # make install clean
...

# it crashed because ther was an old perl installed
root@bsd-test:/usr/ports/lang/perl5.24 # make reinstall
...

# when it was done I went back to the vim port and started the make again
root@bsd-test:/usr/ports/lang/perl5.24 # cd - 
root@bsd-test:/usr/ports/editors/vim # make WITHOUT_X11=yes install clean
...

Now you can use vim :-D

Setup Oh My Zsh on FreeBSD

Thu, 28 Sep 2017 13:33:49 +0200

“Oh My Zsh is an open source, community-driven framework for managing your zsh configuration.

Sounds boring. Let’s try again.

Oh My Zsh will not make you a 10x developer…but you might feel like one.”

First of we need some prerequisites:

pkg
zsh
curl
git

Let’s get this install going.

# become root
su -

# check if pkg is installed if it is not it will promt you with an easy installer.
root@bsd-test:~ # pkg help
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.10.1...
Extracting pkg-1.10.1: 100%
Usage: pkg [-v] [-d] [-l] [-N] [-j <jail name or id>|-c <chroot path>|-r <rootdir>] [-C <configuration file>] [-R <repo config dir>] [-o var=value] [-4|-6] <command> [<args>]
Global options supported:
...
...
...

root@bsd-test:~ # pkg install zsh curl git
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   2.0MB/s    00:03
Processing entries: 100%
FreeBSD repository update completed. 26598 packages processed.
All repositories are up to date.
Updating database digests format: 100%
...
...
...
root@bsd-test:~ #

# now we can set the zsh as the default shell for our user

root@bsd-test:~ # chsh -s zsh svanbroekhoven
chsh: user information updated

# Close the ssh session and reopen it.
# You will be greated with a zsh welcome message, close this with "q"
# Lets pull in oh-my-zsh

bsd-test% sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
Cloning Oh My Zsh...
Cloning into '/home/svanbroekhoven/.oh-my-zsh'...
remote: Counting objects: 831, done.
remote: Compressing objects: 100% (700/700), done.
remote: Total 831 (delta 14), reused 777 (delta 10), pack-reused 0
Receiving objects: 100% (831/831), 568.74 KiB | 988.00 KiB/s, done.
Resolving deltas: 100% (14/14), done.
Looking for an existing zsh config...
Using the Oh My Zsh template file and adding it to ~/.zshrc
         __                                     __
  ____  / /_     ____ ___  __  __   ____  _____/ /_
 / __ \/ __ \   / __ `__ \/ / / /  /_  / / ___/ __ \
/ /_/ / / / /  / / / / / / /_/ /    / /_(__  ) / / /
\____/_/ /_/  /_/ /_/ /_/\__, /    /___/____/_/ /_/
                        /____/                       ....is now installed!


Please look over the ~/.zshrc file to select plugins, themes, and options.

p.s. Follow us at https://twitter.com/ohmyzsh.

p.p.s. Get stickers and t-shirts at http://shop.planetargon.com.

➜  ~

That’s all, now find yourself a fancy theme and place it in: ~/.zshrc

The default is ZSH_THEME=robbyrussell, I often use ZSH_THEME="aussiegeek"

To view the results of your edit simply reopen your ssh session.

Reduce tty count on FreeBSD

Thu, 28 Sep 2017 12:55:27 +0200

After installing FreeBSD I noticed lots of tty’s open that I’ll probably never use so lets disable them with this small script:

#!/bin/sh
sed -i '' -e's|ttyv0.*|ttyv0	"/usr/libexec/getty Pc"		xterm	on  secure|' /etc/ttys
sed -i '' -e's|ttyv1.*|ttyv1	"/usr/libexec/getty Pc"		xterm	on  secure|' /etc/ttys
sed -i '' -e's|ttyv2.*|ttyv2	"/usr/libexec/getty Pc"		xterm	off  secure|' /etc/ttys
sed -i '' -e's|ttyv3.*|ttyv3	"/usr/libexec/getty Pc"		xterm	off  secure|' /etc/ttys
sed -i '' -e's|ttyv4.*|ttyv4	"/usr/libexec/getty Pc"		xterm	off  secure|' /etc/ttys
sed -i '' -e's|ttyv5.*|ttyv5	"/usr/libexec/getty Pc"		xterm	off  secure|' /etc/ttys
sed -i '' -e's|ttyv6.*|ttyv6	"/usr/libexec/getty Pc"		xterm	off  secure|' /etc/ttys
sed -i '' -e's|ttyv7.*|ttyv7	"/usr/libexec/getty Pc"		xterm	off  secure|' /etc/ttys

Reboot your machine and voila, less ttys filling up your ps list. ;-)

Of course you can edit the /etc/ttys file by hand if you like.

I originally found this script on the FreeBSD Forum.

Top like interface for lsof

Tue, 26 Sep 2017 21:32:44 +0200

If you’re curious which pids are using loads of open files this script displays just that.

Example:

The following overview will update +/- every second.

      OPEN / LIMIT      NAME
     32568 / 4096       chrome          (PID  1365)
     14473 / 4096       spotify         (PID  1071)
     12530 / 4096       nylas           (PID  8430)
     10952 / 4096       skypeforlinux   (PID 26476)
      9471 / 4096       skypeforlinux   (PID 26513)
      5360 / 4096       Nylas_Mail_defa (PID  8914)
      4998 / 4096       gnome-shell     (PID   823)
      3726 / 4096       Nylas_Mail_work (PID  8473)
      3528 / 4096       spotify         (PID  1222)
      3105 / 4096       Enpass          (PID  1065)
      2898 / 4096       chrome          (PID  1450)
      2864 / 4096       Nylas_Mail_empt (PID  6152)
      2752 / 4096       tracker-extract (PID  1076)
      2528 / 4096       telegram-deskto (PID 29313)
      1908 / 4096       TeamViewer.exe  (PID 17517)
      1480 / 4096       nylas           (PID 27110)
      1480 / 4096       nylas           (PID 27099)
      1480 / 4096       nylas           (PID 27092)
      1132 / 4096       nylas           (PID  8458)
       890 / 4096       spotify         (PID  1207)
       870 / 4096       gsd-media-keys  (PID  1000)
       858 / 4096       TVGuiSlave.64   (PID 17658)
       845 / 4096       skypeforlinux   (PID 26501)
       810 / 4096       TVGuiDelegate   (PID 17659)
       810 / 4096       solaar          (PID  1070)

lsof-top

#!/bin/bash
# It is pretty clear this is a crude way to do it, and yes it started as a quick one liner.


while true; do
  RESULT=$(
    lsof -n 2>/dev/null | awk '{print $2}' | sort | uniq -c | sort -nr | head -25 | while read nr pid ; do
    printf "%10d / %-10d %-15s (PID %5s)\n" $nr $(cat /proc/$pid/limits | grep 'open files' | awk '{print $5}') $(ps -p $pid -o comm= | sed 's/\ /_/g' ) $pid
    done
  )
  clear
  echo "      OPEN / LIMIT      NAME"
  echo -n "$RESULT"
  sleep 1
done

How it works:

list all open files
clean the lines just keep the pid
count the occurences of pid in the open file list
create a line for the top 25 open file pids in a RESULT var
clear the screen
echo the RESULT

Quick and easy self signed ssl certificate

Fri, 22 Sep 2017 22:46:01 +0200

In many environments you can use ssl encryption, and you may skip the encryption because generating a cert or getting a lets encrypt cert would add more time to the thing your are working on. To overcome this security issue I wrote a small script to create a crt and key file.

selfsigned.sh

#!/bin/bash

# fast ssl cert without ca

ENDPOINT=$1

openssl genrsa -out $ENDPOINT.key 2048
openssl req -new -x509 -key $ENDPOINT.key -out $ENDPOINT.crt -days 3650 -subj /CN=$ENDPOINT

Usage:

 ./selfsigned.sh domain.tld
Generating RSA private key, 2048 bit long modulus
..............+++
....+++
e is 65537 (0x010001)

This created 2 files domain.tld.key and domain.tld.crt

Multiple flavours of nginx http to https redirects

Thu, 21 Sep 2017 00:36:25 +0200

Redirects in nginx are really powerful, here are some cool and common examples.

all domains http > https and remove www

Redirect all http traffic which has no other better matching server_name defined from www.domain.tld and domain.tld to https://domain.tld

# redirect http://www to https non www                                                                               
# global and maybe not the best in all setups
server {
   listen 80;
   server_name ~^(www\.)(?<domain>.+)$;
   return 301 https://$domain$request_uri;
}

http > https

Redirect all of the incoming http requests to https for selected domains

server {
        listen 80;
        server_name domain.tld www.domain.tld;
        return 301 https://$host$request_uri;
}

http > https with exceptions

Redirect all or a part of the incoming http requests to https, you can use this the other way round to redirect a small part of the traffic to https and localion / to the normal backend.

server {
    listen 80;
    server_name domain.tld *.domain.tld;

    # Excluded location from https redirect
    # I use this for lets encrypt validation
    location ~ acme-challenge {
        root        /etc/letsencrypt/temp;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

More will be added ;-)

If you have a redirect you just can’t get to work, leave a comment so I can give it a try.

Nginx map subdomain to subfolder

Wed, 20 Sep 2017 20:56:35 +0200

In this tutorial I’ll show you how to direct your incoming web traffic dynamically to folders on your server. This is really useful for a lot of setups I use this one a lot for development environments because you can give developers rights to create new folders and it automatically has its own sub domain.

Example:

DNS endpoint	Folder on disk
`example.com`	`/var/www/example.com/htdocs/`
`test.example.com`	`/var/www/example.com/test/htdocs/`
`dev1.example.com`	`/var/www/example.com/dev1/htdocs/`

Nginx config

Place this in your server block and remove the current server_name

server_name     "~^(?<subdomain>.+).example.com$";
root            /var/www/example.com/$subdomain/htdocs;

That’s all, if you have questions or suggestions, post them bellow.

Use a deafult cert for nginx

Wed, 26 Apr 2017 15:11:18 +0200

If you use a service like cloudflare or sucuri you don’t need a valid cert on the backend servers to do full ssl to backend. If you want full ssl to the server you need 2 things

self signed cert
nginx proxy to send traffic to port 80

Using this method you need your application to send out redirects to ssl version of pages.

Self signed crt

You can gererate one with: selfsigned.sh

Usage: ./selfsigned.sh server.name.tld

This creates 2 files server.name.tld.key and server.name.tld.crt that should be coppied to /etc/nginx/ssl

Nginx server block

# /etc/nginx/conf.d/000_ssl.conf

server {
  listen 443 default_server ssl;

  ssl_certificate	/etc/nginx/ssl/server.name.tld.crt;
  ssl_certificate_key	/etc/nginx/ssl/server.name.tld.key;

  location / {
      proxy_set_header          Host $host;
      proxy_set_header          X-Forwarded-Proto $scheme;
      proxy_pass                http://localhost:80;
  }

}

Wi-Fi based dual relay on the cheap

Tue, 28 Mar 2017 01:28:42 +0200

I recently discovered the ESP8266 module, it comunicates over serial and you can flash it with custom firware like the arduino bootloader. The module is insanely cheap and realy versatile due to the replacable boot code.

Nagging about the old setup

The reason I started this project is the need for a more reliable relay connected to my domoticz home management system. Before this I was using 433Mhz based relays, at first just bit banging commands over to a 433Mhz transmitter and later with a RFXtrx433E. When I got any of these to work I was just relieved it did as expected and pretty happy with the results. After a while though the cracks started to show, everything has a noticeable delay and you cant send out commands too fast after each other or recievers will simply not react.

Solution Time

MQTT
Wi-Fi
small
cheap

I did a bunch of other projects based around MQTT pub sub systems and know for a fact if is fast, reliable and posible to run on a embedded devices.

I found a open source project that uses those techniques together with domoticz or openhab.

Later more on software and config

My first home made wifi based multi relay

Partlist

In the last part of this spost I’ll list some of the hardware items used to create this device.

Part	Function	Qty needed	Cheapest offer	Seen at
ESP8266	Wi-Fi + microcontroler	1	€1.69	Aliexpress
Breakout board	2mm -> 2.54mm pins	1	€0.16	Aliexpress
2.54mm stackable pin headers	Socket for the ESP	4x 8pins	€0.24	Aliexpress
SRD-05VDC-SL-C T73-5V 10A	High voltage relay	2	€0.60	Aliexpress

more to come this page will be updated in the future

Update forked git repo from original source

Wed, 22 Mar 2017 09:41:49 +0100

1. Clone your fork:

git clone git@github.com:YOUR-USERNAME/YOUR-FORKED-REPO.git

2. Add remote from original repository in your forked repository:

cd into/cloned/fork-repo
git remote add upstream git://github.com/ORIGINAL-DEV-USERNAME/REPO-YOU-FORKED-FROM.git
git fetch upstream

3. Updating your fork from original repo to keep up with their changes:

git pull upstream master

Original post

Map client header to upstream in nginx

Mon, 20 Mar 2017 12:00:22 +0100

Sometimes you want to route traffic to different endpoints based on a http header.

Avoiding if statements in the nginx config is the most important part here.

The example below is based on the user-agent header that is send from the client.

Test nginx conf to view traffic flow

# create upstreams
# android and other non ios device
upstream android {
     server 127.0.0.1:8080;
}
# ios based devices
upstream ios {
     server 127.0.0.1:8081;
}

# map to different upstream backends based on user-agent header
map $http_user_agent $mobile_upstream {
     default    "android";
     ~iPod      "ios";
     ~iPad      "ios";
     ~iOS       "ios";
     ~iPhone    "ios";
}

server {
    listen 80;
    access_log off;
    location / {
        proxy_pass http://$mobile_upstream;
    }
}
server {
    listen 8080;
    access_log /var/log/nginx/default-android.log combined;

}
server {
    listen 8081;
    access_log /var/log/nginx/android-ios.log combined;
}

Another example, when you already have if in your upstream config inside you server{} block


# map function needs to before the server{} block 
# I use:  /etc/nginx/conf.d/00-map-mobile.conf
# set $mobile_device to 0 or 1 
map $http_user_agent $mobile_device {
  default      0;
  ~iPod        1;
  ~iPad        1;
  ~iOS 1;
  ~iPhone      1;
  ~mobile      1;
  ~Android     1;
}

# In the server{} block
include /etc/nginx/includes/ab-upstreams.conf
proxy_pass http://$upstream;

# Inside ab-upstreams.conf
# stuur naar nieuwe locatie       
if ($http_cookie ~ "version=new") {     
        set $upstream "new_cluster";      
} 
if ($request ~ "version=new") { 
        set $upstream "new_cluster";      
} 
  
# stuur user terug naar oude locatie      
if ($http_cookie ~ "version=old") {     
        set $upstream "old_cluster";        
} 
if ($request ~ "version=old") { 
        set $upstream "old_cluster";      
} 
  
# force modbile naar oude upstreams       
if ($mobile_device = 1) { 
        set $upstream "old_cluster";      
}

Push ssh public key to lxc container

Mon, 20 Mar 2017 10:03:42 +0100

LXC is by far my favorite virtualization software for the desktop. I can spin up a base image to test something within seconds.

After starting there is one thing I’ll always do: setup SSH key login, then it feels like any other machine to manage.

Lets go trough the steps.

Setup a test machine

svanbroekhoven@PC-stein:~$ lxc image list
+-------+--------------+--------+-------------------------------------------+--------+----------+------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                DESCRIPTION                |  ARCH  |   SIZE   |         UPLOAD DATE          |
+-------+--------------+--------+-------------------------------------------+--------+----------+------------------------------+
|       | 70223d3f415c | no     | ubuntu 16.04 LTS amd64 (daily) (20170317) | x86_64 | 145.34MB | Mar 18, 2017 at 4:44am (UTC) |
+-------+--------------+--------+-------------------------------------------+--------+----------+------------------------------+
svanbroekhoven@PC-stein:~$ lxc launch ubuntu:16.04 nginx-test
Creating nginx-test
Starting nginx-test          
svanbroekhoven@PC-stein:~$ lxc list
+----------------+---------+----------------------+-----------------------------------------------+------------+-----------+
|      NAME      |  STATE  |         IPV4         |                     IPV6                      |    TYPE    | SNAPSHOTS |
+----------------+---------+----------------------+-----------------------------------------------+------------+-----------+
|   nginx-test   | RUNNING | 10.87.183.245 (eth0) | fd99:3066:2539:9f7d:216:3eff:feef:cefd (eth0) | PERSISTENT | 0         |
+----------------+---------+----------------------+-----------------------------------------------+------------+-----------+

Transfer SSH public key

Now we have a machine running and are ready to push our SSH key into the container.

# Transfer the public key
svanbroekhoven@PC-stein:~$ lxc file push ~/.ssh/id_rsa.pub nginx-test/root/.ssh/authorized_keys  

# Open up a bash shell (ip from the lxc list)
svanbroekhoven@PC-stein:~$ lxc exec nginx-test bash
root@nginx-test:~# ll
total 20
drwx------  3 root root 4096 Mar 20 08:37 ./
drwxr-xr-x 22 root root 4096 Mar  8 02:20 ../
-rw-r--r--  1 root root 3106 Oct 22  2015 .bashrc
-rw-r--r--  1 root root  148 Aug 17  2015 .profile
drwx------  2 root root 4096 Mar 20 08:37 .ssh/

# Edit the file permissions
root@nginx-test:~# chmod 600 /root/.ssh/authorized_keys && sudo chown root: /root/.ssh/authorized_keys
root@nginx-test:~# exit

# Test the connection
svanbroekhoven@PC-stein:~$ ssh root@10.87.183.245
The authenticity of host '10.87.183.245 (10.87.183.245)' can't be established.
ECDSA key fingerprint is SHA256:LA31gxLK9zhA7qMaGwpNNKXcYb+9eht5FPvNwRUM/cE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.87.183.245' (ECDSA) to the list of known hosts.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@nginx-test:~# logout
Connection to 10.87.183.245 closed.
svanbroekhoven@PC-stein:~$

Comodo PositiveSSL CA Chain

Fri, 17 Mar 2017 15:04:01 +0100

This is the current PositiveSSL ca chain.

Apache and Nginx seem to have different implementations on the chain order these are the right orders for PositiveSSL.

Nginx

Apache

Centos auto start service

Fri, 17 Mar 2017 13:37:09 +0100

There are many ways to accomplish the same result this is just my preference.

Centos 6

To auto start services in Centos or Redhat OS, you can use builtin chkconfig utility. It is located in /sbin directory. If you are a regular user (non-root), then /sbin may not be in your path. Therefore, you may have to use the full path to access the chkconfig utility.

To auto start a new service: Find out the name of service script from /etc/init.d/ directory e.g. mysqld or httpd Add it to chkconfig

sudo /sbin/chkconfig --add mysqld

Make sure it is in the chkconfig.

sudo /sbin/chkconfig --list mysqld

Set it to autostart

sudo /sbin/chkconfig mysqld on

To stop a service from auto starting on boot

sudo /sbin/chkconfig mysqld off

Centos 7

When a service you installed needs to be started at boot use the following:

sudo systemctl enable nginx.service

Apache force www subdomain

Fri, 17 Mar 2017 13:25:23 +0100

A redirect in an apache vhost to force the website to www.$domain.tld

<VirtualHost *:80>

...

RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}%{REQUEST_URI} [R=302,L]


# test and reload
apachectl -t && apachectl -k graceful

Backup all databases on a plesk server

Fri, 12 Aug 2016 10:55:00 +0200

This is not the fastest not the safest and not the most eficient, but it does work. and unless you have a realy bussy database this is a pretty oke way to backup your database. It uses mysqldump and the internal plesk credentials so no need to type passwords at all.

It can ofcourse be used on any other mysql server as long as you set the right credentials.

#! /bin/bash
# aapjeisbaas.nl
# 2016-08-12
 
TIMESTAMP=$(date +"%F")
BACKUP_DIR="/tmp/db-dump/$TIMESTAMP"
MYSQL_USER="admin"
MYSQL=/usr/bin/mysql
MYSQL_PASSWORD="$(cat /etc/psa/.psa.shadow)"
MYSQLDUMP=/usr/bin/mysqldump 

mkdir -p "$BACKUP_DIR"

databases=$($MYSQL --user=$MYSQL_USER -p$MYSQL_PASSWORD -e "SHOW DATABASES;" | grep -Ev "(Database|information_schema|performance_schema)")
for db in $databases; do
  $MYSQLDUMP --force --opt --user=$MYSQL_USER -p$MYSQL_PASSWORD --databases $db | gzip > "$BACKUP_DIR/$db.sql.gz"
done

Put this in a file: vim backup-db.sh make it executable: chmod +x backup-db.sh and run it ./backup-db.sh

This will take some time, in some cases 15 minutes or even more. It will backup to: /tmp/db-dump/$TIMESTAMP

Keep nginx from pushing backend port on try and redirects

Fri, 05 Aug 2016 12:10:00 +0200

Ever found yourself wondering why your nginx setup is returning a port from the backend vhost?

QUICK FIX: port_in_redirect off;

I found some tweaks to prevent this from happening, first a example:

# This is the result of a: try $uri $uri/ in a backend vhost
stein@stein ~ $ curl --head http://www.example.nl/test
HTTP/1.1 301 Moved Permanently
Server: nginx
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: http://www.example.nl:8080/test/
Accept-Ranges: bytes
X-Varnish: 1044342160
Age: 0
Via: 1.1 varnish
Vary: User-Agent
X-Cache: MISS

The nginx config I used before ifound out what was wrong is:

server {
        listen                          80;
        server_name                     *.example.nl example.nl;

        root                            /var/www/vhosts/example.nl/httpdocs;
        include                         includes/default_vhost.conf; # proxy timeouts etc. nothing interesting

        location / {
                include                         includes/proxy_varnish.conf; 	# location with varnish related conf
	}									# varnish sends requests on to 8080 if not in cache
}
server {
        listen                          8080;
        server_name                     *.example.nl example.nl;

        root                            /var/www/vhosts/example.nl/httpdocs;
        index                           index.php index.html index.htm;

        location / {
                index                           index.php;
                try_files                       $uri $uri/ @handler;
        }

        location ~ .php/ {
                rewrite                         ^(.*.php)/ $1 last;
        }

        location @handler {
                rewrite                         / /index.php;
        }

        location ~ \.php/ {
                rewrite ^(.*\.php)/ $1 last;
        }

        include includes/hhvm.conf; # Sends fascgi to hhvm
}

It frustrated me that the solution is not that well documented but here it is: port_in_redirect off;

Yes, noting exiting here, just place port_in_redirect off; in the backend vhost. There is also a way to not send the full hostname: server_name_in_redirect off;

This will result in a config like this:

server {
        listen                          80;
        server_name                     *.example.nl example.nl;

        root                            /var/www/vhosts/example.nl/httpdocs;
        include                         includes/default_vhost.conf; # proxy timeouts etc. nothing interesting

        location / {
                include                         includes/proxy_varnish.conf; 	# location with varnish related conf
	}									# varnish sends requests on to 8080 if not in cache
}
server {
        listen                          8080;
        server_name                     *.example.nl example.nl;

        root                            /var/www/vhosts/example.nl/httpdocs;
        index                           index.php index.html index.htm;
	port_in_redirect		off; 

        location / {
                index                           index.php;
                try_files                       $uri $uri/ @handler;
        }

        location ~ .php/ {
                rewrite                         ^(.*.php)/ $1 last;
        }

        location @handler {
                rewrite                         / /index.php;
        }

        location ~ \.php/ {
                rewrite ^(.*\.php)/ $1 last;
        }

        include includes/hhvm.conf; # Sends fascgi to hhvm
}

Connect chromebook to openvpn with custom routes

Sat, 23 Jul 2016 19:17:00 +0200

This post will try to help you get a openvpn connection on a chromebook with a pushed route instead of a redirected gateway. By default chromeos redirects all traffic to the vpn. To overcome this you can make a onc file that can use all the ovpn config options instead of the very limited gui options.

To start you will need the following files:

ca.crt
client.key
client.crt
client.ovpn (or equivalent)
# user and pass auth enabled in server (auth-user-pass)

At the end you will have 3 files that will allow you to connect to the vpn.

ca.crt
client.p12
chrome-vpn.onc

First we create the p12 file to be able to import the user cert and key combo into the chromeos device. This can be done on a external system or in a shell on the chromebook in developer mode. Puting your device in developer mode is not necessary if you have a separate machine to create the. p12

# After running this command you will be asked to set a password for the p12.
# You will need this password to read the contents of it. (when you import it)
openssl pkcs12 -export -in client.crt -inkey client.key -certfile ca.crt -name ChromeBook -out client.p12

Import the root CA user p12 on your device

Go to the folowing URL on your chromebook: chrome://settings/certificates Click on the Authorities tab Import the ca.crt and select "Trust this certificate for identifiying websites."

Go to the first tab in the certificate manager (Your certificates) Click on “Import and bind to device” Select client.p12 Click on OK and a password window will appear. Input the password you chose earlier when you created the p12 file

That was the easy part, now for creating the onc file. Underneath is an example:

{
 "Type":"UnencryptedConfiguration",
      "Certificates": [ {
      "GUID": "{<GUID#1>}",
      "Type": "Authority",
      "X509": "<CA_CERT>"
   } ],
    "NetworkConfigurations": [ {
      "GUID": "{<GUID#2>}",
      "Name": "<VPN_NAME>",
      "Type": "VPN",
      "VPN": {
          "Type": "OpenVPN",
          "Host": "<HOSTHAME>",
          "OpenVPN": {
                        "ServerCARef": "{<GUID#1>}",
                    "AuthRetry": "interact",
                    "ClientCertType": "Pattern",
                    "ClientCertPattern": {              
                          "IssuerCARef": [ "{<GUID#1>}" ]
                         },
                    "CompLZO": "true",
                    "Port": 1194,
                    "Proto": "udp",
                    "RemoteCertTLS":"server",
                    "RemoteCertEKU": "TLS Web Server Authentication",
                    "SaveCredentials": false,
                    "ServerPollTimeout": 10,
                    "Username": "<USERNAME>",
                    "KeyDirection":"1",                    
                    "TLSAuthContents":"<TLS_AUTH_KEY>"
                     }
             }
                               } ]
}

In there we first replace the <GUID> with a guid from: https://www.uuidgenerator.net/ Replace all the <GUID#1> with the same guid, use another one for the <GUID#2>

Change the <CA_CERT> with the content of the ca.crt file without: -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----

Example:

# ca.crt:
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----


#becomes one line without begin and end:
"X509": "MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOF.....KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg=="

Rename the VPN connection, this is how will sea it from the user perspective. Chenge <VPN_NAME> to whatever you want.

Now We get to the actual OpenVPN config

Edit the folowing: <HOSTHAME> to your vpn server hostname or ip

Make sure that these are the same uuid as the one in the ca crt block:

"ServerCARef": "{<GUID#1>}",
"IssuerCARef": [ "{<GUID#1>}" ]

Then we have some options:

"CompLZO": "true",
"Port": 1194,
"Proto": "udp",
"RemoteCertTLS":"server",
"RemoteCertEKU": "TLS Web Server Authentication",
"SaveCredentials": true,
"ServerPollTimeout": 10,
"Username": "<USERNAME>",
"KeyDirection":"1",

You can add a password with: "Password": "supersectretpass",

Remove or add whatever you need, you can find the full spec here ONC spec

If you use tls auth with your vpn there is a catch in making te config. You need to strip the # comment lines and place \n in the place of newlines. Here is what is should look like:

"TLSAuthContents":"-----BEGIN OpenVPN Static key V1-----\n1f2537daea1be955f7...ef396b1c5df8f5a8c\n-----END OpenVPN Static key V1-----\n"

Now for the option that makes youable to only use pushed routes from the vpn server instead of redirecting everything:

"IgnoreDefaultRoute": true,

Put that in the openvpn block to disable the automatic redirection of all your web traffic. Here is my Office onc file with all the “juicy” details removed.

You can save the file and import it here: chrome://net-internals/#chromeos

{
 "Type":"UnencryptedConfiguration",
      "Certificates": [ {
      "GUID": "{90a39a1e-2ce8-4fc9-b824-0e1a4faad3e5}",
      "Type": "Authority",
      "X509": "MIIEMjCCAxqgAw......eNDl6nIw=="
   } ],
    "NetworkConfigurations": [ {
      "GUID": "{c44e6e82-c0c2-44e8-9f1c-579404464dc4}",
      "Name": "Office VPN",
      "Type": "VPN",
      "VPN": {
          "Type": "OpenVPN",
          "Host": "office.domain.tld",
          "OpenVPN": {
                        "ServerCARef": "{90a39a1e-2ce8-4fc9-b824-0e1a4faad3e5}",
                    "AuthRetry": "interact",
                    "ClientCertType": "Pattern",
                    "ClientCertPattern": {              
                          "IssuerCARef": [ "{90a39a1e-2ce8-4fc9-b824-0e1a4faad3e5}" ]
                         },
                    "Port": 1194,
                    "Proto": "udp",
                    "Cipher": "AES-256-CBC",
                    "CompLZO": "true",
                    "IgnoreDefaultRoute": true,
                    "RemoteCertTLS":"server",
                    "SaveCredentials": true,
                    "ServerPollTimeout": 10,
                    "Username": "user",
                    "Password": "secretpass",
                    "KeyDirection":"1",                    
                    "TLSAuthContents":"-----BEGIN OpenVPN Static key V1-----\n1f2537da....a8c\n-----END OpenVPN Static key V1-----\n"
                     }
             }
                               } ]
}

For more info check the folowing links:

ONC Spec ChromeOS VPN ONC block example

Auto update wordpress

Fri, 22 Jul 2016 16:41:00 +0200

Neat trick to always auto update wordpress and plugins.

Pro Tip: You should make regular backups to have something to go back to if the update f*d your site ;-)

First we need to change the permissions and ownership of the web dir.

# Change Owner and Group to SomeUser and WebServerGroup (ftp-user:www-data)
chown -R ftp-user:www-data /path/to/wp
# Change folders to 775 
find /path/to/wp -type d -exec chmod 775 {} \; 
# files op 664 
find /path/to/wp -type f -exec chmod 664 {} \;

Now we need to set the prefered file access method for wordpress.

Open the wp-config.php file and add the following lines:

define('FS_METHOD', 'direct');
define( 'WP_AUTO_UPDATE_CORE', true );

This update uses a MU plugin (Must Use plugin)

This can be achiefed by creating the following directory: wp-content/mu-plugins

All php files in this dir will allways be used by wordpress.

Now create wp-content/mu-plugins/updates.php and add the following content:

add_filter( 'allow_dev_auto_core_updates', '__return_false' ); 
add_filter( 'auto_update_plugin', '__return_true' ); 
add_filter( 'auto_update_theme', '__return_true' );

Now your site will stay up to date ;-)

Remote diff over ssh

Wed, 13 Jul 2016 14:00:00 +0200

Want to do a diff on a local and remote file, or on 2 remote files.

Here is a small trick

# Local and remote diff
 diff file <(ssh remote 'cat file')

# Or both are on remote servers
diff <(ssh remote1 'cat file') <(ssh remote2 'cat file')

I use this a lot when a ansible –check run tells me there is a diff and want to know what I will be overwriting.

Run a script when load is too high

Mon, 04 Jul 2016 10:00:00 +0200

This is a small script that can check if server load is too high and start another script.

It is NOT best practice to fix a problem like this but it can get you trough the nights while developers work on the code problem. I called my script /usr/local/bin/damnnastycheck.sh it runs every minute from /etc/crontab yhe only requirement is BashCalc (bc)

#!/bin/bash
high=8.1

if [ $(echo $(cat /proc/loadavg | awk '{print $1}')'>'$high | bc -l) == "1"  ]; then
  echo "high load, running fix"
  /usr/local/bin/damnnastyfix.sh
fi

In the damnnastyfix you could place something like:

#!/bin/bash
killall -9 httpd
service httpd restart

Make iptables like your ftp sessions

Wed, 06 Apr 2016 23:10:00 +0200

If you manage your firewall by hand have issues with connection trough ftp, try adding this.

# FTP Helper (beginning of script)
modprobe ip_conntrack_ftp
modprobe ip_nat_ftp ports=21

# ... 
# other rules
# ...

# FTP Helper
iptables -A OUTPUT -o eth0 -p tcp --sport ftp -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp --sport ftp-data -j ACCEPT

# Drop rule 

#(end of script)

Find broken RAM DIMM in server

Mon, 14 Mar 2016 15:10:00 +0100

A broken ram module is pretty annoying in itself but trying to find it by trial and error is even worse. This simple command shows the locations of the active dimms in the system. Run it and find the missing one, in this case:

CPU#0Channel#0_DIMM#1

[root@machine ~]# cat /sys/devices/system/edac/mc/mc*/csrow*/ch*_dimm_label 
CPU#0Channel#0_DIMM#0
CPU#0Channel#1_DIMM#0
CPU#0Channel#2_DIMM#0
CPU#0Channel#1_DIMM#1
CPU#0Channel#2_DIMM#1
CPU#1Channel#0_DIMM#0
CPU#1Channel#1_DIMM#0
CPU#1Channel#2_DIMM#0
CPU#1Channel#0_DIMM#1
CPU#1Channel#1_DIMM#1
CPU#1Channel#2_DIMM#1
[root@machine ~]#

Or:

[root@machine ~]# cat /sys/devices/system/edac/mc/mc*/dimm*/dimm_label

Block tor traffic in cloudflare firewall

Wed, 09 Mar 2016 18:00:00 +0100

![tor logo] ({filename}/static/images/tor.png)

If you don’t want to allow access to your server through the tor network you can ask nicely or just add every malicious looking client to a list. I tried to come up with a better solution.

I started with a script that blocked incomming connections on a loadbalancer (can also be used on a webserver) Then I realized that if you use cloudclare in front of that the tcp connections come from cloudare and not from the tor endpoints. So I needed to rebuild it to add the rules to the cloudflrae firewall.

I started of the same way, create a variable which contains all the tor ip adresses. Then It get’s a bit more complex, working with a json api and bash is often not without struggle. The code is commented, if that is not enough contact me and I will try to make i more readable. Use it, cron it, love it ;-)


#!/bin/bash

echo "Tor endpoint list loading"
TORLIST=$(curl -s https://check.torproject.org/exit-addresses |grep ExitAddress | awk '{print $2 }' | sort | uniq)

ZONE=""   # can be left empty this script will pick the first one from your account for you
KEY=""    # API key can be found at: https://www.cloudflare.com/a/account/my-account
EMAIL=""  # email adress used to login to cloudflare

# If the ZONE is not defined get the first one from the api
if [ -z "$ZONE" ]; then
  ZONE=$( curl -s -X GET "https://api.cloudflare.com/client/v4/zones/" \
  -H "Content-Type:application/json" \
  -H "X-Auth-Key:${KEY}" \
  -H "X-Auth-Email:${EMAIL}" | python -m json.tool |grep -A 1 development_mode | grep id | awk '{print $2}' | sed 's/"//g' |sed 's/,//g')
  echo "Add zone string to config for faster result and less api calls"
  echo ${ZONE}
fi

echo "Retrieving active firewall rules"
IP_LIST=""
# The api is paged so we first qeury the api and check how many apges we have
PAGES=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/${ZONE}/firewall/access_rules/rules?mode=block&configuration_target=ip&page=1&per_page=1000&order=scope_type&direction=desc&match=all " \
  -H "Content-Type:application/json" \
  -H "X-Auth-Key:${KEY}" \
  -H "X-Auth-Email:${EMAIL}" | python -m json.tool |  grep  total_pages | awk '{print $2}')

# for every page gat the response rinse out the bullshit keep the ip adresses
# Because we can have multiple pages I cocatinate the srings into IP_LIST
for ((page=1; $page <= ${PAGES}; page++)); do
  DATA=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/${ZONE}/firewall/access_rules/rules?mode=block&configuration_target=ip&page=${page}&per_page=1000&order=scope_type&direction=desc&match=all " \
  -H "Content-Type:application/json" \
  -H "X-Auth-Key:${KEY}" \
  -H "X-Auth-Email:${EMAIL}" | python -m json.tool |  grep value | awk '{print $2}')

  IP_LIST="${IP_LIST} ${DATA}"
done

# Now we have a list with all the tor ip exit nodes, and all the ip adresses in the firewall.
# I creatd the next loop to take all the tor endpoints and check if they are already in the firewall.
# If the tor ip is not found in the firewall list it will be added to the firewall
# There is a sleep in the loop to keep the api from blocking your reqeusts (max 1200 per 5 min.)

for ip in ${TORLIST}; do
  sleep 0.3

  IN_FIREWALL=$(echo ${IP_LIST} | grep ${ip})

  if [ -z "$IN_FIREWALL" ]; then
    echo "adding ${ip} to blocklist"
    curl -s -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE}/firewall/access_rules/rules" \
    -H "Content-Type:application/json" \
    -H "X-Auth-Key:${KEY}" \
    -H "X-Auth-Email:${EMAIL}" \
    --data '{"mode":"block","configuration":{"target":"ip","value":"'"${ip}"'"},"notes":"This site is not available through tor endpoints"}' > /dev/null
# else
#   echo "already in blocklist"
  fi
done

Block almost all tor traffic to your server

Wed, 09 Mar 2016 01:15:00 +0100

![tor logo] ({filename}/static/images/tor.png)

If you don’t want to allow access to your server through the tor network you can ask nicely or just add every malicious looking client to a list. I tried to come up with a better solution.

This Script takes the known tor endpoints from torproject.org and adds it to a ipset list. The ipset is the dropped with iptables.

#!/bin/bash

echo "Tor endpoint list loading"
TORLIST=$(curl -s https://check.torproject.org/exit-addresses |grep ExitAddress | awk '{print $2 }' | sort | uniq)

echo "creating ipset tor list"
ipset destroy torset
ipset -N torset iphash

for ip in ${TORLIST}; do
    ipset -A torset ${ip}
done
iptables -A INPUT -m set --match-set torset src -j DROP

Copy mysql database to dev location

Mon, 01 Feb 2016 11:10:00 +0100

![db copy] ({filename}/static/images/db_copy.jpg)

If you are just here for the script copy the script and have fun. If you want a bit more info read on this script is tweakable to some extend.

#! /bin/bash
 
TIMESTAMP=$(date +"%F")
BACKUP_DIR="/tmp/db-dump/$TIMESTAMP"
MYSQL_USER="root"
MYSQL=/usr/bin/mysql
MYSQL_PASSWORD="password"
MYSQLDUMP=/usr/bin/mysqldump

mkdir -p "$BACKUP_DIR"

databases="live_db"
for db in $databases; do
  test_db=$db"_test"
  $MYSQLDUMP --user=$MYSQL_USER -p$MYSQL_PASSWORD $db | gzip > "$BACKUP_DIR/$db.sql.gz"
  mysql --user=$MYSQL_USER -p$MYSQL_PASSWORD -e "DROP DATABASE IF EXISTS $test_db;"
  mysql --user=$MYSQL_USER -p$MYSQL_PASSWORD -e "CREATE DATABASE $test_db;"
  zcat "$BACKUP_DIR/$db.sql.gz" | mysql --user=$MYSQL_USER -p$MYSQL_PASSWORD $test_db
done

rm -rf "$BACKUP_DIR"

This script works single threaded and can take in multiple databases or even all databaeses and copy them to _test databases. If you want it to copy all databases to _test use this line for defining the databases var:

databases=$($MYSQL --user=$MYSQL_USER -p$MYSQL_PASSWORD -e "SHOW DATABASES;" | grep -Ev "(Database|information_schema|performance_schema)")

The biggest issue with this method is that it is slow and you may want to consider skipping disk and pass output of the dump straight into the import with the following:

  test_db=$db"_test"
  mysql --user=$MYSQL_USER -p$MYSQL_PASSWORD -e "DROP DATABASE IF EXISTS $test_db;"
  mysql --user=$MYSQL_USER -p$MYSQL_PASSWORD -e "CREATE DATABASE $test_db;"
  $MYSQLDUMP --user=$MYSQL_USER -p$MYSQL_PASSWORD $db | mysql --user=$MYSQL_USER -p$MYSQL_PASSWORD $test_db

There is a downside to this method, while you are copying the database your data in the destination location is longer unavailable. The database is dropped and then filled as it comes in from the dump instead of being parsed at max read speed of disk or max execute speed of mysql.

The last method is realy good for a seperate dev mysql server but can fill up your ram if the dev mysql server can not keep up with the dump speed.

Let's Encrypt

Tue, 15 Dec 2015 22:17:00 +0100

![Let’s encrypt] ({filename}/static/images/letsencrypt.svg)

This Article is all about the SSL cert signing and auto signing for Let’s Encrypt. I only cover nginx based setup and does require root and knowledge of some basic nginx vhost configuration.

Let’s start with what Let’s Encrypt is and how to use it. It is a public CA that signs for free, there are some limits but reaching them is not likeley in most cases. To Get a valid cert they use their own application that is developed in tho open on this github page.

Installing the client is pretty simple:

user@server:~$ git clone https://github.com/letsencrypt/letsencrypt
user@server:~$ cd letsencrypt
user@server:~/letsencrypt$ ./letsencrypt-auto --help

Now that you have the client you can request a first cert:

./letsencrypt-auto certonly -a webroot --webroot-path=/var/www/your/web/root -d test.yourdomain.com

For this to work you need the have:

the domain test.yourdomain.com going to the dir /var/www/your/web/root
write permisions in the folder
allow for the following location to be reached from a external server http://test.yourdomain.com/.well-known/acme-challenge/

The last point is vital for checking if you are the owner of the domain and if you are like me and 403 everything starting with “.” then this will fail and you need to work around it in your vhost conf.

I tackle this all in one simple nginx include:

user@server:~$ cat /etc/nginx/includes/ssl-validate.conf
location ~ acme-challenge {
    root        /etc/letsencrypt/temp;
}

This sets the root dir for anything that looks like a acme-challenge to: /etc/letsencrypt/temp This needs to be included in the top of every port 80 vhost you want to have this work for.include /etc/nginx/includes/ssl-validate.conf;

That was half of the equation the “webroot” part of it now we need to get some domains in a list, a script to request cert and a cronjob to do this automnomous.

First you need to put the repo checkout in a sensible place:

user@server:~$ cd /usr/local/bin/
user@server:~$ git clone https://github.com/letsencrypt/letsencrypt

Then we need a script to check and manage the certs:

user@server:~$  cat /usr/local/bin/ssl-renew
#vhosts=$(ls -lah /var/log/nginx/*access*log |awk '{print $9}' | sed 's/\/var\/log\/nginx\///g' | sed 's/.access.log//g' |grep -ve '*\|access.log' |uniq |sort -h)

vhosts="
test.yourdomain.com
test.your-other-domain.com
"

export DIR=/etc/letsencrypt/temp
mkdir -p $DIR

clear
for i in $vhosts
do
  echo "-------------------------------------------------------------------------------------------"
  echo $i

  if openssl x509 -checkend 604800 -noout -in /etc/letsencrypt/live/$i/cert.pem
  then
    echo "Certificate is good for at least another week"
  else
    echo "Certificate has expired or will do so within 7 days!"
    echo "(or is invalid/not found)"
    echo "starting renew procedure"
    /usr/local/bin/letsencrypt/letsencrypt-auto --renew certonly -a webroot --webroot-path=$DIR -d $i
    /usr/sbin/nginx -t && /usr/sbin/nginx -s reload
  fi

done

As you can see there are 2 ways of loading the vhosts in to the loop from a simpje list of from the names in the nginx access logs my filenames look like: some-subdomain.aapjeisbaas.nl.access.log

For now just stick with the list if you need more flexibility you cat try the nginx access log trick.

This script checks the cert is in the default location and if it is valid for at least a week. If one of those things is not true it will request if it is not found or renew a cert. (for extra security remove --renew to generate a new key for every cert.)

Now chmod the file, run it and check output debug where needed. If all goes well add it to a cron file:

user@server:~$  cat /etc/cron.d/ssl-check 
7 2 * * * root  /usr/local/bin/ssl-renew | /sbin/sendmail -F mail@domain.com -t mail@domain.com

The certs are in /etc/letsencrypt/live/ every domain has their own folder. Here is a example nginx conf for the test domain.

user@server:~$ cat /etc/nginx/conf.d/test.domain.com.conf
server {
    listen      80;
    listen      [::]:80;
    server_name test.domain.com;
    include     /etc/nginx/includes/ssl-validate.conf;
    location / {
        return      302 https://test.domain.com/;
    }
server {
    listen      443;
    listen      [::]:443;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/test.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/test.domain.com/privkey.pem;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:DHE-RSA-AES128-SHA:DHE-RS-AES256-SHA:EDH-RSA-DES-CBC3-SHA:ECDHE:DHE:HIGH:!MD5:!aNULL:!EDH:!RC4;
    ssl_prefer_server_ciphers on;
    ssl_stapling on;
    resolver your.closest.dns.resolver; # maybe use 8.8.8.8 if you want


    server_name test.domain.com;

    include     /etc/nginx/includes/log.conf;

    root        /var/www/vhosts/test.domain.com;

    location / {
      add_header "Cache-Control" $cacheable_types;
    }
}