AWS re:Invent 2016 Second Keynote: We Are All Transformers

In addition to this post, please also click here to read my AWS re:Invent Tuesday Night Live with James Hamilton recap and here to read my AWS re:Invent 2016 first keynote recap from Wednesday.

After a whirlwind of product announcements from CEO Andy Jassy the previous day, it was time for Werner Vogels, CTO of Amazon Web Services, to take the stage. You can view the keynote in its entirety below. You can also read on to get a digest of Vogel’s keynote along with links to get more information about the announced new services.

Sporting a Transformers t-shirt, Vogels talked about AWS’s role in helping to bring about IT transformation. He very specifically addressed users, particularly developers, about their role as transformers in the places where they worked. AWS can do this, explained Vogels, because they have strived from the very beginning to be the most customer centric IT company on Earth.

Screen Shot 2016-12-01 at 11.35.05 AM.png

To meet their goal of making their customers transformers in their businesses, Vogels talked about three ways that AWS can help create transformers.

Screen Shot 2016-12-01 at 11.51.50 AM.png

In the area of development, Vogels emphasized the importance of code development and testing because that’s where users can experiment and where businesses can be agile.

Screen Shot 2016-12-01 at 11.54.25 AM.png

To help users transform the way they do development and testing, Vogels focused on AWS products, old and new, that help bring about operational excellence, particularly in the areas of preparedness, operations and responsiveness.

Screen Shot 2016-12-01 at 11.59.59 AM.png

In the area of preparing, Vogels talked about the importance of automating as many tasks as possible in order to build reliable, secure and efficient development, test and production environments. A key service to enable automation on AWS is CloudFormation and although no new announcements were made in this area, Vogels took some time to review the new features that have been added to CloudFormation in 2016.

Screen Shot 2016-12-01 at 12.03.25 PM.png

Many customers make use of Chef cookbooks to prepare and to configure their AWS environments. This is in large part because of AWS OpsWorks which is a configuration management service based on Chef Solo. Taking this to the next step, Vogels announced a new AWS OpsWorks for Chef Automate service. This new service provides a user with a fully managed Chef server to removed one more operational burden they had to contend with previously. You can read more about AWS OpsWorks for Chef Automate here.

screen-shot-2016-12-01-at-12-04-45-pm

Moving on to systems management, Vogels announced Amazon EC2 Systems Manager, which is a collection of AWS tools to help with mundane administration tasks such as packaging, installation, patching, inventory, etc.. You can read more about AWS EC2 Systems Manager here.

Screen Shot 2016-12-01 at 12.05.37 PM.png

Transitioning to operating as the next area of operational excellence transformation, Vogels made the argument that code development and continuous integration/continuous deployment is a crucial part of operations. After reviewing the existing services that AWS has to assist users with making the code development process more agile, Vogels announced AWS CodeBuild to go with the existing CodeCommit, CodeDeploy and CodePipeline services.

screen-shot-2016-12-06-at-4-49-51-pm

AWS CodeBuild is a fully managed service that automates building environments using the latest checked-in code and running unit tests again that code. This service streamlines the development process for users and reduces the risk of errors. You can read more about AWS CodeBuild here.

screen-shot-2016-12-01-at-12-08-29-pm

Another key aspect of operating is monitoring. As he had done previously, Vogels reviewed the existing services that help users gain visibility into their environments.

Screen Shot 2016-12-01 at 12.11.13 PM.png

Taking the next step to help users gain deeper insights into how their applications are running, Vogels harkened back to Jassy’s keynote theme of superpowers to introduce AWS X-Ray. Acknowledging the difficulty of debugging distributed systems, AWS released X-Ray to give users the ability to trace requests across their entire application and to map our the relationship between various services in the system. This insight makes it easier for developers to troubleshoot and to improve their applications. You can read more about AWS X-Ray here.

screen-shot-2016-12-01-at-12-13-40-pm

The final area of operational excellence Vogels covered was responding. How can users respond to errors and alarms and do so in an automated fashion that can also escalate issues in a timely fashion when necessary?

screen-shot-2016-12-01-at-12-16-00-pm

One answer from AWS is the new AWS personal Health Dashboard. Based off the existing AWS Service Health Dashboard, this new service provides users with a personalized view of the system health of AWS. The new dashboard will show the performance and availability of services that is being accessed by a user. Users will also receive alerts that are triggered by a degradation in the services that are being leveraged by them and users can write Lambda functions to respond to those events. You can read more about AWS personal Health Dashboard here.

techjournalist_2016-dec-01-2

AWS and their customers also have to respond to security issues. Distributed Denial of Service attacks have been the top threat for web applications with many different types of attacks at different layers of the networking stack. Historically, most of these DDoS attacks have tended towards Volumetric and State Exhaustion attacks.

Screen Shot 2016-12-06 at 5.47.50 PM.png

To address these attacks, Vogels announced AWS Shield. This is a managed service that works in conjunction with other AWS services like Elastic Load Balancing and Route 53 to protect user web applications. AWS Shield comes in two flavors – AWS Shield Standard and AWS Shield Advanced. Standard is available to all AWS customers at no extra cost and protects users from 96% of the most common attacks.

screen-shot-2016-12-06-at-5-50-38-pm

AWS Shield Advanced provides additional DDoS mitigation capability for volumetric attacks, intelligent attack detection, and mitigation for attacks at the application & network layers. Users get 24×7 access to the AWS DDoS Response Team (DRT) for custom mitigation during attacks, advanced real-time metrics and reports, and DDoS cost protection to guard against bill spikes in the aftermath of a DDoS attack. You can read more about AWS Shield Standard and AWS Shield Advanced here.

screen-shot-2016-12-06-at-5-54-27-pm

Transitioning away from transforming operational excellence, Vogels moved to transformation though using data as a competitive differentiator. Because of the cloud, Vogels asserted, everyone has access to services such as data warehousing and business intelligence. What will differentiate companies from each will other will be the quality of the data they have and the quality of the analytics they perform on that data.

The first new service announcement that Vogels made in this area was AWS Pinpoint. Pinpoint is a service that helps users run targeted campaigns to improve user engagement. It uses analytics to help define customer target segments, send targeted notifications to that target segment and track how well a particular campaign did. You can read more about AWS Pinpoint here.

awsreinvent_2016-dec-01-7

Moving on, Vogels argued that 80% of analytics work is not actually analytics but hard work to prepare and to operate an environment where you can actually do useful queries of your data. AWS is on a mission to flip this so 80% of analytics work done by users will actually be analytics.

screen-shot-2016-12-01-at-12-53-39-pm

Vogels argued that AWS already has number of services to address most of the  work that falls into that 80% bucket. To address even more of that 80%, Vogels introduced a new service called AWS Glue. Glue is a data catalog and ETL service that simplifies movement of data between different AWS data stores. It also allows users to automate tasks like data discovery, conversion, mapping and job scheduling. You can read more about AWS Glue here.

Screen Shot 2016-12-07 at 9.29.07 AM.png

By adding AWS Glue, Vogels argued that AWS now has all the pieces required to build the industry’s best modern data architecture.

Screen Shot 2016-12-01 at 1.05.23 PM.png

Another need for users in the is space, said Vogels, is large-scale batch processing which normally requires a great deal of heavy lifting to set up and use. To help here, Vogels announced AWS Batch. Batch is a managed service that lets users do batch processing without having to provision, manage, monitor, or maintain clusters. You can read more about AWS Batch here.

techjournalist_2016-dec-01-3

The last area of transformation Vogels addressed took him back to the roots of AWS – Compute. Except of course, “compute” at AWS has grown beyond Elastic Compute and virtual machines. Vogels reminded the audience that AWS compute has now grown to also include containers with Elastic Container Service and Serverless/Function as a Service with Lambda.

kongyang_2016-dec-01

Since all the new announcements about compute was made by Jassy in the previous keynote, Vogels focused on the containers and Lambda parts of their compute spectrum. For users of ECS, Vogels previewed a new task placement engine which will give users finer-grain control over scheduling policies.

Screen Shot 2016-12-07 at 2.04.53 PM.png

Beyond this, Vogels acknowledged that customers have requested the flexibility to build their own custom container schedulers to work with ECS or to integrate with existing schedulers such as Docker Swarm, Kubernetes or Mesos. To enable this, Vogels announced that AWS is open sourcing Blox, a collection of open source projects for building container management and orchestration services for ECS.

Screen Shot 2016-12-01 at 1.25.28 PM.png

The first two components of Blox will be a cluster state service for handling event streams that came from ECS and the second component will be a daemon-scheduler that will help launch daemons in container instances. You can read more about Blox here.

Screen Shot 2016-12-01 at 1.25.38 PM.png

Moving on to the last compute area, Vogels talked about serverless/Lambda. Lambda already supported a number of languages and AWS added to that list by adding support for C#.

awsreinvent_2016-dec-01-14

Vogels then mentioned that one the most frequent requests they receive from uses is the ability to execute tasks at the edge of the AWS content delivery network instead of having to go back to a source further away and incurring unwanted extra latency. To address this request, Vogels announced AWS Lambda@Edge. This new service can inspect HTTP requests and execute Lambda functions at CloudFront edge locations when appropriate. You can read more about AWS Lambda@Edge here.

Screen Shot 2016-12-01 at 1.40.13 PM.png

Finally to coordinate multiple Lambda functions in a simple and reliable manner, Vogels announced AWS Step Functions. This service gives users the ability to visually create a state machine which specifies and executes all the steps of a Lambda application. A state machine defines a set of steps that performs work, makes decisions, and controls progress on Lambda functions. You can read more about AWS Step Function here.

Screen Shot 2016-12-01 at 1.42.28 PM.png

Wrapping up his keynote, Vogels summarized all the product announcements that had been made during his and Jassy’s keynotes.

Screen Shot 2016-12-07 at 4.23.56 PM.png

With that, Vogels ended his keynote with a charge to the audience to use all the tools they have been given to go and transform their businesses.

AWS re:Invent 2016 First Keynote: Andy Jassy Is Your Shazam

In addition to this post, please also click here to read my AWS re:Invent Tuesday Night Live with James Hamilton recap and here to read my AWS re:Invent 2016 second keynote recap from Thursday.

shazamdc6

I grew up watching a TV show called Shazam! which was based on a comic I also read by the same name. The main protagonist was a superhero called Captain Marvel, who was given his superpowers by a wizard named Shazam. Captain Marvel used the power of Shazam to fight evil and to help save the human race.

At the first keynote for AWS re:Invent 2016, Andy Jassy, CEO of Amazon Web Services, played the part of the wizard who could give everyone cloudy superpowers as he wrapped the keynote around the theme of superpowers. You can view the keynote in its entirety below. You can also read on to get a digest of Jassey’s keynote along with links to get more information about the announced new services.

To set the table, Jassy started the keynote with a business update before giving what everyone in attendance and tuning in was waiting for – a litany of new AWS features and capabilities.

Screen Shot 2016-11-30 at 11.02.45 AM.png

Amazon Web Services continues to grow at an astounding rate with no let up in sight. It is by far the fastest growing billion dollar enterprise IT company in the world, suggesting that it is a safe choice for enterprises.

Screen Shot 2016-11-30 at 11.05.59 AM.png

And the growth is not just coming from startups anymore but includes a growing stable of enterprise customers.

Screen Shot 2016-11-30 at 11.03.22 AM.png

While the keynote included something for everyone, Jassy clearly had new enterprise customers in mind as he walked through the value proposition for AWS, explained basic AWS services, unveiled new services and directed his ire at Larry Ellison and Oracle. And to frame the rest of his keynote, Jassy assumed his Shazam wizard persona and explained what AWS can do for customers to give them cloudy superpowers.

werner_2016-nov-30-7

The first superpower theme to be highlighted was supersonic speed and how AWS enables customers to move more quickly. This not only refers to customers being able to launch thousands of cloud instances in minutes but the ability to go from conception to realization of an idea by taking advantage of all the many services that AWS has to offer.

Screen Shot 2016-11-30 at 11.10.22 AM.png

While AWS already boasts more services than any other cloud provider, Jassy pointed out that their pace of innovation has been increasing to the rate of 1,000+ new features or significant capabilities rolled out in 2016. That equates to an average of 3 new capabilities added per day.

Screen Shot 2016-11-30 at 11.12.53 AM.png

Continuing the focus on supersonic speed, Jassy followed with announcements about new EC2 instance types to add to the already burgeoning compute catalog. In particular, updates to four instance type families, to meet varying compute use cases, were announced.

Screen Shot 2016-12-03 at 5.13.04 PM.png

Two new extra-large instance types were added to the T2 family which doubled and quadrupled respectively the large instance type. T2 instances are suited for general purpose workloads that require occasional bursting and the extra large instances give users more bang for their buck while providing even more burst capacity. You can read about the new T2 instance types here.

Screen Shot 2016-12-03 at 5.04.17 PM.png

For memory intensive workloads, a new R4 instance type was announced which effectively doubled the capabilities of the previous R3 instance type. This memory-optimized instance type is suitable for any workload that benefits most from in-memory processing. You can read more about the new R4 instance type here.

Screen Shot 2016-12-03 at 5.08.00 PM.png

A new I3 instance type was introduced that is optimized for I/O intensive workloads. This new instance type will use SSDs to increase IOPS capabilities by orders of magnitude over the current I2 instance type. The I3 will be ideally suited for transaction oriented workloads such as databases and analytics. You can read more about the new I3 instance type here.

Screen Shot 2016-11-30 at 11.29.30 AM.png

Next up was the new C5 compute-optimized instance type using the new Intel Skylake CPU. The C5 will be suitable for workloads that require CPU-intensive workloads such as machine learning and financial operations requiring fast floating point calculations. You can read more about the new C5 instance type here.

Screen Shot 2016-11-30 at 11.30.31 AM.png

Another area where speed is important are computational workloads that require a Graphic Processing Unit (GPU) to offload processing from the CPU. Jassy announced that AWS is working on a feature called Elastic GPUs For EC2. This will allow GPUs to be attached to any instance type as workload demands require, similar in concept to Elastic Block Storage. You can read more about the Elastic GPUs here.

Screen Shot 2016-11-30 at 11.32.33 AM.png

The last new instance type to be announced was the F1 instance type utilizing customizable FPGAs which will give developers the flexibility to program these instances to meet specific workload demands in a way that could not be done with standard CPUs. You can read more about the new F1 instance type here.

Screen Shot 2016-11-30 at 11.37.06 AM.png

Accelerating how fast users can move goes beyond new hardware and new instance types. There is also the need to simplify complex tasks whenever possible. Cloud providers like Digital Ocean have carved out a strong niche market by specializing in offering no-frills Virtual Private Servers (VPS). A VPS is a low-cost hosted virtual server that is designed to be easy for users to set up and suitable for running applications that do not have high performance requirements.

AWS is taking VPS providers like Digital Ocean head on with their new Amazon Lightsail service. For as little as $5 a month , users can launch new instances in their VPC and do so by walking through minimal configuration steps.

Screen Shot 2016-11-30 at 11.34.41 AM.png

Behind the scenes, Lightsail will create a VPS preconfigured with SSD-based storage, DNS management, and a static IP address. As underscored below, all the steps in the box are performed on behalf of the user. You can read more about Amazon Lightsail here.

Screen Shot 2016-11-30 at 11.35.30 AM.png

Moving on the next superpower that AWS can give users, Jassy talked about x-ray vision and how it can benefit cloud users. The first benefit was mainly a not so subtle dig at Larry Ellison and Oracle and other legacy vendors.

Screen Shot 2016-12-03 at 11.20.12 PM.png

Jassy’s argument was that on AWS, users can run their own tests and benchmarks on true production like environments instead of accepting the word of untrustworthy vendors. It was one of many negative attacks on Oracle during Jassy’s keynote.

Getting back on point, Jassy talked about the benefit for users of being able to perform business analytics on the data they’ve uploaded to AWS as part of the x-ray vision power that AWS gives to them. Jassy then highlighted the breadth of the existing AWS services for doing analytics to help users better understand their customers.

Screen Shot 2016-11-30 at 11.41.43 AM.png

Enhancing this portfolio, Jassy unveiled a new service called Amazon Athena. Athena is a new query service for analyzing stored S3 data using standard SQL. In essence, users can treat their S3 as a data lake and perform queries against unstructured data to unearth actionable intelligence. You can read more about Amazon Athena here.

Screen Shot 2016-11-30 at 11.45.22 AM.png

Another benefit of “x-ray vision” which Jassy presented was the ability for users to see meaning inside their data through artificial intelligence. Jassy pointed out that Amazon, the parent company, has been leveraging artificial intelligence and deep learning for their own businesses.

Screen Shot 2016-12-04 at 9.51.07 PM.png

Naturally, AWS is leveraging the learnings and tools of Amazon to create a suite of new services focused on artificial intelligence called Amazon AI.

Screen Shot 2016-11-30 at 11.52.12 AM.png

The first service in the suite is Amazon Rekognition for image recognition and analysis. This service is powered by deep learning technology developed inside Amazon that is already being used to analyze billions of images daily. Users can leverage Rekognition to create applications for use cases such as visual surveillance or user authentication. You can read more about Amazon Rekognition here.

Screen Shot 2016-11-30 at 11.52.40 AM.png

Moving from image to voice AI, Jassy next introduced Amazon Polly, a service for converting text to speech. Polly initially supports 24 different languages and can speak in 47 different voices. Powered also by deep learning technology created by Amazon, Polly can translate text that may have ambiguous meanings by understanding the context of the text. User can leverage Polly to create applications that require all types of computer generated speech. You can read more about Amazon Polly here.

Screen Shot 2016-11-30 at 11.54.25 AM.png

Rounding out the new AI suite, Jassy introduced Amazon Lex for natural language understand and for voice recognition. Based on the same deep learning technology behind Alexa, which powers the Amazon Echo, users can build Lex based applications such as chatbots or anything that supports conversational engagement between humans and software. You can read more about Amazon Lex here.

screen-shot-2016-11-30-at-11-55-59-am

Another superpower trumpeted by Jassy was that of flight, which he used as a metaphor for having the freedom to build fast, to understand data better and most importantly, to escape from hostile database vendors. To incentivize users to leave their traditional database vendors, AWS had previously introduced a Database Migration service and the Amazon Aurora MySQL-Compatible database service. As it turned out, enterprises liked Aurora but also wanted support for PostgreSQL. So Jassy took this opportunity to announced a new Amazon Aurora PostgreSQL-Compatible database service.

Screen Shot 2016-11-30 at 12.48.04 PM.png

This new service uses a modified version of the PostgreSQL database that is more scalable and has 2x the performance of the open source version of PostgreSQL but maintains 100% API compatibility. You can read more about PostgreSQL for Aurora here.

Screen Shot 2016-11-30 at 12.48.21 PM.png

The last superpower discussed by Jassy was shape-shifting, which was another metaphor, this time for AWS’ ability to integrate with on-premises infrastructures. To kick off this section of the keynote, Jassy revisited an announcement that had been made previously of a joint service called VMware Cloud on AWS. This service is simply a managed offering, running on AWS, that supports VMware technologies such as vSphere, vSAN and NSX. You can read more about VMware Cloud on AWS here.

Screen Shot 2016-12-05 at 12.00.00 AM.png

Then in perhaps a somewhat tortured attempt to keep to the current theme, Jassy tried to expand the meaning of on-premises infrastructure beyond servers in the data center to sensors and IoT devices.

Screen Shot 2016-12-05 at 12.11.32 AM.png

Making the transition to talking IoT services, Jassy discussed the challenges of running device on the edge of the network in order to collect and to process data from these sensors and growing number of IoT devices.

awsreinvent_2016-nov-30-10

To help address these challenges, Jassy announced their new AWS Greengrass service which embeds AWS services like Lambda in field devices. Manufacturers can OEM Greengrass for their devices and users can leverage Greengrass to collect date in the field, process the date locally and forward them to the cloud for long-term storage and further processing. You can read more about AWS Greengrass here.

awsreinvent_2016-nov-30-11

Of course, any discussion about on-premises infrastructure by AWS ultimately leads back to their desire to move all on-premises workloads to what they consider the only true cloud – AWS. So perhaps it’s no surprise that Jassy would wrap up his keynote with two solutions for expediting the migration of data to AWS.

During the last re:Invent in 2015, AWS announced the Snowball which is a 50 TB appliance for import/export of data to and from AWS. As these Snowball appliances have been put to use, customers have expressed a desire for additional capabilities such as local processing of data on the appliance. To facilitate these new capabilities, Jassy announced the new Amazon Snowball Edge.

Screen Shot 2016-11-30 at 1.14.04 PM.png

The Snowball Edge adds more connectivity, doubles the storage capacity, enables clustering of two appliances, adds new storage endpoints that can be accessed from existing S3 and NFS clients and adds Lambda-powered local processing. You can read more about the AWS Snowball Edge here.

Screen Shot 2016-11-30 at 1.14.31 PM.png

Going back to the enterprise and rounding out the keynote, Jassy asked the question, “What about for Exabytes (of data)?” The answer, Jassy proposed, is a bigger box. Then in a demonstration of showmanship worthy of any legacy vendor, out came the new Amazon Snowmobile.

Screen Shot 2016-12-05 at 12.54.36 AM.png

The proposition of the Snowmobile is very simple. Enterprises will be able to move 100 PBs of data at a time so that an exabyte-scale data transfer that would take ~26 years to do over a 10 Gbps dedicated connection can be completed in ~6 months using Snowmobiles. You can read more about the AWS Snowmobile here.

Screen Shot 2016-12-05 at 12.55.41 AM.png

The spectacle of the Snowmobile being driven on stage proved to be an appropriate capper to the morning keynote with Andy Jassy’s turn as the superpower-giving wizard, Shazam.