My approach at making AWS EC2 ~80% cheaper: Automatic replacement of Autoscaling nodes with equivalent spot instances

Note

AutoSpotting, the tool described here, was since open sourced and is now available on GitHub.

This post merely states the development history of the tool, it is seriously outdated, and only kept here for historical reasons. It was written in April 2016 and it is describing the state of the tool at that moment. Please refer to the GitHub page for more up-to-date information about the current state of the project.

Getting started

Last year, during one of the sessions of the Berlin AWS meetup where I am often present, during the networking that happened after the event @freenerd from Mapbox mentioned something about the spot market, saying how much cheaper it is for them to run instances there, but also the fact that for their use case it sometimes happened that the instances were terminated in the middle of their batch processing job that prepares the map for the entire world.

A few weeks later, at another session of the AWS meetup, I participated in a similar discussion where someone mentioned the possibility to have instances attached to an on-demand AutoScaling group, which was a feature just released by AWS at that time. I don’t remember if spot was mentioned in the same discussion, or if it was all in my mind, but somehow these concepts got connected and I thought this is a nice problem to hack on.

I was thinking about the problem for a while, and after a couple of weeks I came up with an algorithm based on the instance attach/detach mechanism supported by AutoScaling. I tested it manually and I quickly confirmed that AutoScaling happily allows attaching spot instances and detaching on-demand ones in order to keep the capacity constant, but that it often tries to rebalance the availability zones, so in order for it not to interfere with the automation, the trick is to try to keep the group more or less balanced across availability zones, so that AutoScaling won’t try to rebalance it.

I soon started coding a prototype in my spare time, which is actually my first non-trivial program written in a while, and to make it even more interesting, I chose to write it in golang.

Slow progress

After a few weeks of coding, in which I rewrote it at least twice(and even now I’m still nowhere near being happy with how it looks), I realized it’s quite a bit harder and more complex than I initially thought. Other things happened and I kind of lost interest, I stopped working on it and it all got stuck.

A few months later at the re:invent conference I attended some talks where I met some other folks interested by this problem and I saw other approaches of attacking the problem, with multiple AutoScaling groups, and that was also when I first got in touch with someone from SpotInst who was trying to promote their solution and was sharing business cards.

After re:invent I became a bit more active for a while, I also tried to get some collaborators but failed at it, so I kept working on it in my spare time every now and then and I got closer to get it work. Then I recently had a long vacation, and immediately after I returned I attended the Berlin AWS Summit, where I met the SpotInst folks once again, and it seems they now have a full fledged solution based on pretty much a reimplementation of AutoScaling as a SaaS, they are apparently successful with it. This motivated me to work even harder on this, since my solution is simpler, cleaner and just as effective as theirs.

Breakthrough

After the Berlin AWS Summit, having my batteries charged, I resumed my work and after a few coding nights I managed to make my prototype work. It took much longer than expected, but at least I got there, yay! 🙂

What I have so far

  • An easy to install CloudFormation template that creates an SNS topic, a Lambda function written in golang(with a small JS wrapper that downloads and run it), subscribed to the topic and a few IAM settings to make it all work (update: this was largely simplified since)
  • A golang binary, for now closed source(update: not anymore), but I’m going to open it up once I get it in a good enough shape so that I’m not ashamed of it and after I get all the approvals from my corporate overlords, who according to my employment contract need to approve the publishing of such non-trivial code

 

How does it work

The lambda function is executed by a custom CloudFormation resource when creating the CloudFormation stack from the template, and it subscribes to both your topic and a topic that I run, which fires it every 30 minutes, using a scheduled event.

When my scheduled function runs the lambda function, it will concurrently inspect the AutoScaling groups from all the AWS regions and it will ignore all those that are not tagged with the EC2 tags it expects.

The AutoScaling groups marked with the expected tag will be processed concurrently, on each of them gradually replacing the on-demand instances with compatible spot instances, one at a time. Each run will either launch a single spot instance or attach a launched spot instance to the AutoScaling group, after detaching an on-demand one it is meant to replace. The spot instance is not attached while its uptime is less than the Autoscaling group’s grace period.

The spot instance bid price matches the price of the on-demand instance it is meant to replace. If your spot request is outbid, AutoScaling will handle it as a regular instance failure, and will immediately replace it with an on-demand instance. That instance will later be replaced by the cheapest available compatible spot instance, likely of a different type and with a different spot price.

In practice the group should converge to the most stable instance pricing, no the long term saving about 80% from the normal on-demand EC2 price.

How to use it/Getting started

All you need to do is set an EC2 tag on the AutoScaling group where you want to test it. Any other AutoScaling groups will be ignored.

The tag should have the following attributes:

Key: “spot-enabled”

Value: “true”

See my next blog post for a full installation and runtime walkthrough where you can see very detailed instrunctions on how to get started.

Feedback is more than welcome

If you find any bugs or you would like to suggest any improvements, please get in touch on gitter or file an issue on GitHub.

Warning

This is experimental, summarily tested and likely full of bugs, so you should not run it on production, but it should be safe enough for evaluation purposes.

Anyway, use it at your own risk, and don’t hold me responsible for any misuse, bugs or damage this may cause you.

Update: many of these issues were ironed out since and the tool is currently stable enough for production use cases. It is already in use in dozens of companies, where it is already generating considerable savings. Feel free to also give it a try and report your feedback on gitter.

Later Updates

  • 27 Apr 2016
    • if you want to see it in action, please also check out my next blog post which walks you in detail through the installation and setup process, also explaining the currently known issues and their workarounds as of the time of the writing of this post.
  • 5 May 2016
    • bug fixes since the previous update
      • no longer spinning the AutoScaling group when running at minimum capacity
      • increased runtime frequency to once every 5min to make it converge faster
    • currently known issues
      • when first enabling it on a group with multiple on-demand nodes, it sometimes may launch extra spot instances that do not get added to the AutoScaling group(up to as much as the initial size of the group). workaround: terminate them manually from the AWS console. They will not be re-launched
      • spinning condition when the AutoScaling group is set to a fixed size (Min=Max). Workaround: set Max to Min+1 and disable any scaling actions you may have configured, in order to keep the group at the minimum capacity
    • things currently being worked on
      • fixes for the known issues mentioned above
      • major under-the-hood code refactoring in preparation of open sourcing
      • choosing instance types that are unlikely to be terminated in the near future, based on historical stats data sourced from the Spot Bid Advisor
      • mark spot instances as protected from termination by AutoScaling
      • if you have any other suggestions please write a comment below.
  • 10 July 2016
    • new features
      • improved algorithm for picking the new spot instance type
        • always launch a new spot instance  from the same zone of an existing on-demand instance. Previously the zone was the one where we had the less instances, which often may have been one where we had no running instances and no way . This was causing the bugs about spinning and launch of additional spot instances that were not added to the group.
        • allow multiple spot instances of a given type for each availability zone, as long as their total number is less than 20% of the total capacity from the group. For example a group of 15 instances using 3 availability zones will allow for 3 identical instances per availability zone, but the fourth instance from a zone will be of a different instance type.
      • Lambda wrapper updates
        • rewrote the Lambda wrapper in Python, which makes it more maintainable, since I’m much better at Python than at JavaScript
        • Implement versioning for the binary blob, by downloading the latest version only if not already there, based on the content SHA hash
      • CloudFormation cleanup
        • remove the SNS topic that was never used
      • support the new Mumbai region(still needs testing)
      • internal code refactoring
        • lots of cleanups that make it more maintainable
        • improved logging
    • bug fixes since the previous update
      • it should no longer start additional spot instances, the final capacity should match the original on-demand capacity, unless there were any AutoScaling actions.
      • fixed spinning condition with fixed-size AutoScaling groups by temporarily increasing the group during the replacement process
      • fix choosing of the cheapest compatible/redundant enough spot instance type, previously any cheaper instance type may have been chosen, not necessarily the cheapest.
      • lots of other small bugfixes for various edge cases
    • things currently being worked on
      • open sourcing process was started, and I already got some of the required approvals
      • figuring out how to implement automated testing
    • backlog
      • choosing instance types that are unlikely to be terminated in the near future, based on historical stats data sourced from the Spot Bid Advisor
      • mark spot instances as protected from termination by AutoScaling
  • 13 July 2016
    • Mostly bugfixes, many thanks to @nmeierpolys for some very valuable bug reports, fast feedback and a lot of patience while testing it.
      • improved conversion of on-demand launch configuration fields into spot launch configuration equivalents. In addition to user_data, SSH keys, EBS volumes and many other mostly trivial to convert fields that were previously handled, the following more complex fields should now also be better handled, which make it work on much more real-life environments:
        • EC2 Classic security groups
        • detailed instance monitoring
        • associating public IP addresses
      • It was successfully tested on complex EC2-classic and VPC setups where many of these fields were being used.
      • Compatibility notice: In the likely event that you are using IAM roles on your instances, you need to update to the latest version of the CloudFormation template, since the launch of such spot instances would otherwise fail due to missing IAM permissions required to run instances set up with IAM roles. Again thanks to @nmeierpolys for finding out this issue and proposing the fix.
  • 18 July 2016
    • Improve handling of storage volumes.
      • Bugfix: Fix panic while copying EBS storage configurations.
      • New feature: Implement compatibility check for storage volumes based on the number of attached ephemeral disk volumes present in the launch configuration. For example an instance which has a launch configuration that attached it a couple of the ephemeral SSD drives of a certain size would only be replaced by instance types which provide SSD devices at least as many and of at least the same size each, in order not to violate the storage expectations from the new instances.
  • This is largely outdated, current information about the project can be seen on GitHub

Preparing and Ubuntu image for serial console, on a hard-disk connected over USB

It’s been almost a year since my last post, hopefully I will be able to post more often from now on.

This time I’m making a howto on how to install Ubuntu on a SATA disk-drive while having it connected over USB through an USB2SATA adapter, then how to customize Ubuntu so that all the boot messages and the console are directed to a serial port console.

What Am I trying to do?

You might ask yourselves why would you want to do that… Well, I don’t know about you, but I needed this in order to prepare my coreboot development environment on a motherboard that I will only access over Serial port or SSH. Now a bit of history… I’ve been in Berlin for the last three months as part of a business trip, sent by the notorious Finnish mobile phone company that I am working for. While I was at LinuxTag back in May, I finally met the coreboot developers that I’ve been chatting with on IRC ever since 3 years ago and bought myself a coreboot-supported motherboard (Asrock E350M1) from one of the coreboot developers living in Berlin, Peter ‘CareBear\’ Stuge. I bought it because I’ve been planing for quite a while now to build myself a home computer or set-top-box for my TV back at home, and this board seems to be perfect for the job. As a bonus, it is off course running coreboot and quite hacker-friendly.

The coreboot support for this board is still work in progress and although there are a few rough edges, the motherboard is running pretty well, and booting up very fast (under 1 second to the Grub menu). Still, there are a few problems here and there and as a coreboot developer that I like to say I am, although my contributions to coreboot were minor so far, I would like to help getting this board better supported.

The prefered debugging mecanism of coreboot is the serial console because it’s relatively easy to initialize and pretty common. Unfortunately this board doesn’t provide a console port on the back panel, but it has a header with the required pins somewhere on the PCB.

Yesterday me and Peter spent a lot of time working on this board, trying to build a serial header for it and getting it up to speed for coreboot development. We bought some components and then Peter built a nice serial-to-header adapter that also works ad a NULL-modem serial cable since I didn’t have a proper NULL-modem cable.

Then we tried to get an OS running on the board from a SSD drive, but unfortunately the image we had was not properly set up, so we decided to build a new OS installation.

Hardware Setup

As I said so far, I have the Asrock motherboard, a serial-to USB adapter and the custom serial header adapter made by Peter. Besides these, I also have a laptop and a portable laptop SATA hard-drive with an USB-to-SATA adapter.

Software setup

I chose to do it with Ubuntu because it’s easy to set up, quick to install, and pretty nice for development. The hard-disk was connected over USB and I slready had it partitioned, so I only reused the first partition already created there.

I reformatted the first partition to EXT4.

sudo mkfs.ext4 -L rootfs /dev/sdb1

Ubuntu then mounted the first partition to /media/rootfs after double clicking on it.

Installing the base Ubuntu packaged in there. You can replace the architecture to i386 for a 32bit OS, natty with another Ubuntu release, and choose a mirror closer to you.

sudo debootstrap –arch amd64 natty /media/rootfs http://de.archive.ubuntu.com/ubuntu/

After this is done, we can bind-mount some filesystems from the host, preparing for our chroot into the new Ubuntu install.

sudo mount -o bind /dev /media/rootfs/dev

sudo mount -o bind /proc /media/rootfs/proc

sudo mount -o bind /sys /media/rootfs/sys

And finally, chroot

sudo chroot /media/rootfs /bin/bash

Create some config files in the new system

cat << EOF >  /etc/fstab
# device mount type options freq passno
LABEL=root / ext3 defaults,errors=remount-ro 0 1
LABEL=swap none swap sw 0 0
EOF

echo coreboot > /etc/hostname

Set up networking for DHCP

echo -e “auto eth0 \n iface eth0 inet dhcp” >/etc/network/interfaces

Add “restricted universe multiverse” to the line you should have in /etc/apt/sources.list

Install some vital packages

apt-get install linux-image grub-pc

Serial port configuration for Grub

Open /etc/default/grub with an editor.

Comment out

#GRUB_HIDDEN_TIMEOUT=0

Set

GRUB_CMDLINE_LINUX_DEFAULT=”console=ttyS0,115200″

Add these two lines

GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND=”serial –speed=115200 –unit=0 –word=8 –parity=no –stop=1″

Then you can update the grub configuration.

update-grub

Install grub on the hard-disk

grub-install /dev/sdb

Configure Linux console on the serial port

cat << EOF >  /etc/init/ttyS0.conf
# ttyS0 – getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -L 115200 ttyS0 vt102
EOF

Set a root pasword

passwd

Exit the chroot, unmount all the directories mounted there, connect the hard-disk and the serial cable to the motherboard and enjoy the new OS over the serial console.

~Cristi

Master of Puppets

Hi,

It’s been a long time since my previous post and many good things happened to me ever since. A few months ago I changed my job, moved to a new home, bought a bike, adopted a lovely little cat and today I finally graduated my MSc studies.

Lately I’ve been working on the thesis project, and I’m very glad it’s over. The project implemented a netinstall and configuration management system for a gLite cluster, all written in Puppet and available on my github page

Now that I finished this I can finally spend more time watching Star Trek or hacking on coreboot :), I just got an old RTL8029AS NIC from an ex-colleague (Thanks Serban Cordis!) that I’ll try to get working as a remote debug console in Coreboot and SerialICE just like Rudolf Marek did a while ago.

Cristi

25’th birthday

Hi,
Today I had my 25’th birthday (and Christmas), and we all had a great time together.
I’ve been away for quite a while now, it’s been almost a week with no Internet connection but it seems I was able to survive…
Me and my wife were gone to Berlin for a Rammstein concert, and after driving more than 3000Km in 4 full days, we finally got home. The concert was great, even better than we expected, but the whole trip was very long and tiresome so we had to rest quite a lot after the arrival. Thanks Paula, this was my best birthday present ever!

Merry Christmas to everyone!
Cristi

coreboot @ Budapest meetup

Yesterday I had a presentation in Budapest, at their New Technology Meetup.

It was a short introduction to the coreboot project. I saw some cool projects made by the other presenters and participants, like a Z80-based hand-made computer able to run Basic and a very spectacular DJ application full of graphic and sound effects.

I had a lot of fun, and I can say that it did worth all the effort of driving there and back in a single day for 15 hours or so (never trust a GPS device blindly, you can get into trouble!). I only wish they didn’t talk in Hungarian while presenting so that we could understand the discussions *hint*.

Thanks to the organizers, especially Balazs Fejes and Tamas Terray for inviting me.

Here you can see the recording.

Cristi

Just married

Hello,

Finally, we managed to wake up.. This weekend I celebrated my wedding with Paula, the woman I chose to spend the rest of my life with. These were the happiest and the more exciting days of my life to date. It was a very busy weekend, with lots of things to prepare and bear (the hardest was the Orthodox religious service that took more than an hour), but over all we had a lot of fun in the process.

Sure, there were some glitches, like anytime you have to satisfy a crowd of 120 people with totally incompatible musical tastes (and not only!), but we did our best and over all it all was great and most of them were delighted from what we could tell.

The ring was hardly bearable at first, just like a tooth filling 🙂 but I’m starting to get used to it. The priest made a mistake at first and in the middle of the service I had to insert/extract twice Paula’s ring off both my annular fingers with great effort and help from my godfathers. I’ll try to avoid churches as much as possible from now on, and only enter again in a church if really necessary 🙂

We’re now eagerly preparing for the honey moon^H^H^H^Hweek, and we hope to get some pictures by the time we return.

See you soon,
Cristi&Paula