Andrew Griffiths

Build a Realtime Serverless GraphQL API with Websockets on AWS

Sun, 15 Jul 2018 00:00:00 GMT

With the release of AWS AppSync we finally have the ability to create realtime serverless apps on AWS. Previously you were forced to spin up EC2 instances in order to create websocket connections as they are not supported by AWS Lambda.

In the tutorial you’ll learn how to build a simple GraphQL API using AWS AppSync. Then you’ll write a client app in vanilla javascript (no frameworks) that receives realtime updates via websockets. Let’s get started!

Set Up
Create GraphQL Schema
Create Mapping Templates
Deploy AppSync GratphQL API with Serverless Framework
Create GraphQL Queries For Client Application
Download API Config from AWS Console
Isomorphic Vanilla JavaScript Client Code For Subscribing to Realtime API Updates
Node.js Client Application Build Process
Node.js Client Application
Browser Client Build Process
Browser Client
Conclusion

1. Setup

Go ahead and install the serverless framework cli which we will be using to deploy our AppSync GraphQL API and create a new directory for our project.

$ npm install -g serverless
$ mkdir realtime-chat
$ cd realtime-chat

2. Create GraphQL Schema

We will define a basic GraphQL schema for our chat application.

schema {
    query: Query
    mutation: Mutation
    subscription: Subscription
}

type Subscription {
    inbox(to: String): Page
    @aws_subscribe(mutations: ["message"])
}

type Mutation {
    message(body: String!, to: String!): Page!
}

type Message {
    from: String!
    to: String!
    body: String!
    sentAt: String!
}

type Query {
    me: String
}

A standard GraphQL schema apart from the realtime subscription which uses a special syntax to indicate to AWS AppSync which mutation to subscribe to (@aws_subscribe(mutations: ["message"])), in this case the message mutation.

3. Create Mapping Templates

Now we have our schema defined we need to add resolvers for it. If you’re expecting to need to write a lambda function you’d be wrong! AppSync introduces the concept of mapping templates which translate the client request to one the backing store (DynamoDB, elasticsearch etc) understands and then translates the response back to the client again.

To keep things simple we are creating an API without a database. AppSync offers a special type of resolver called a local resolver which does not persist the request data but instead just relays it on to whatever subscribers exist at the time.

Let’s create a directory to house our mapping templates.

$ mkdir mapping-templates

Then let’s create the request template for our message mutation in a file called mapping-templates/Message.request.vtl which will extract the fields from the mutation request.

{
  "version": "2017-02-28",
  "payload": {
    "body": "${context.arguments.body}",
    "from": "${context.identity.username}",
    "to":  "${context.arguments.to}",
    "sentAt": "$util.time.nowISO8601()"
  }
}

For the response we just use the standard forwarding template. Create a file called mapping-templates/ForwardResult.response.vtl with the following contents.

$util.toJson($context.result)

Your folder structure should now look like the following:

$ tree mapping-templates
mapping-templates
├── ForwardResult.response.vtl
└── Message.request.vtl

4. Deploy AppSync GraphQL API with Serverless Framework

Now we need to create a config file for the serverless framework to provision our API. In order to do this we’re going to use the Serverless-AppSync-Plugin.

Install it with npm.

$ npm install --dev serverless-appsync-plugin

Then create a serverless.yml file with the following contents.

---
service: realtime-chat

frameworkVersion: ">=1.21.0 <2.0.0"

plugins:
  - serverless-appsync-plugin

provider:
  name: aws
  region: eu-west-1

custom:
  awsAccountId: ${env:AWS_ACCOUNT_ID}
  appSync:
    name: realtimeChat
    apiKey: ${env:APPSYNC_API_KEY}
    apiId: ${env:APPSYNC_API_ID}
    authenticationType: API_KEY
    schema: schema/schema.graphql
    serviceRole: "AppSyncServiceRole" # AppSyncServiceRole is a role defined by amazon and available in all accounts
    mappingTemplatesLocation: mapping-templates
    mappingTemplates:
      - dataSource: Chat
        type: Mutation
        field: message
        request: Message.request.vtl
        response: ForwardResult.response.vtl
      - dataSource: Chat
        type: Subscription
        field: inbox
        request: Message.request.vtl
        response: ForwardResult.response.vtl
    dataSources:
      - type: NONE  # use an AppSync local resolver
        name: Chat
        description: 'Chat relay'

As you can see we set the data source type to NONE in order to use the local resolver as we do not want to persist the chat messages in a database but instead just forward them to other clients listening for updates.

Our serverless.yml config contains a few environment variables that we must supply. Let’s create a .env file that contains our AWS Account ID and dynamically populates the other variables.

# .env

export AWS_ACCOUNT_ID=123456789

export APPSYNC_API_ID=$(aws appsync list-graphql-apis \
  --query 'graphqlApis[?name==`realtimeChat`].apiId' \
   --output text >/dev/null 2>&1)

export APPSYNC_API_KEY=$(aws appsync list-api-keys \
  --api-id "$APPSYNC_API_ID" \
  --query 'apiKeys[0].id' \
   --output text >/dev/null 2>&1)

Now we’re ready to deploy our API with a single command:

$ .env && sls deploy

Congratulations! You’ve just deployed a GraphQL API with realtime support.

5. Create GraphQL Queries For Client Application

The next thing we need to do is to create the GraphQL queries that we will be using from our client to query our API.

First let’s create a directory to house our client code.

$ mkdir src

Then let’s create a directory for our queries.

$ mkdir src/graphql

Create a file at src/graphql/inboxSubscription.js for our subscription query with the following contents:

import gql from 'graphql-tag';

export default gql`
subscription Inbox($to: String) {
    inbox(to: $to) {
      from
      body
    }
}`;

This is just a simple subscription query which will return from and body message fields.

6. Download API Config from AWS Console

We need to download the config settings for the app so that it can connect to our GraphQL API.

Navigate to AppSync section in the AWS Console. Select your API and download the web config settings.

You will have a file named AppSync.js in your download area. Move this your src directory and rename it to config.js.

The config file you download should look like the following if you have left the default option for securing your API as API_KEY.

export default {
  "graphqlEndpoint": "https://xxxx.appsync-api.eu-west-1.amazonaws.com/graphql",
  "region": "eu-west-1",
  "authenticationType": "API_KEY",
  "apiKey": "xxxxxxxxxxxxxxxxxxxxxxxxx"
}

7. Isomorphic Vanilla JavaScript Client Code For Subscribing to Realtime API Updates

We are going to create an isomorphic client - one that can be run either in the browser or via node.js in a terminal.

First let’s install the dependencies we’ll need.

$ npm install -s apollo-cache-inmemory apollo-client apollo-link aws-appsync aws-sdk es6-promise graphql graphql-cli graphql-tag isomorphic-fetch ws

Then let’s create an entrypoint for the application.

$ touch src/index.js

Your app source code directory should now have the following contents.

$ tree src
src
├── config.js
├── graphql
│   └── inboxSubscription.js
└── index.js

Paste the following code into your index.js file:

const RECIPIENT = 'Bobby';

if (!global.WebSocket) {
  global.WebSocket = require('ws');
}
if (!global.window) {
  global.window = {
    setTimeout: setTimeout,
    clearTimeout: clearTimeout,
    WebSocket: global.WebSocket,
    ArrayBuffer: global.ArrayBuffer,
    addEventListener: function () { },
    navigator: { onLine: true }
  };
}
if (!global.localStorage) {
  global.localStorage = {
    store: {},
    getItem: function (key) {
      return this.store[key];
    },
    setItem: function (key, value) {
      this.store[key] = value;
    },
    removeItem: function (key) {
      delete this.store[key];
    }
  };
}
require('es6-promise').polyfill();
require('isomorphic-fetch');

// Require config file downloaded from AWS console with endpoint and auth info
const AppSyncConfig = require('./config').default;
const AWSAppSyncClient = require('aws-appsync').default;
import InboxSubscription from './graphql/inboxSubscription';

// Set up Apollo client
const client = new AWSAppSyncClient({
  url: AppSyncConfig.graphqlEndpoint,
  region: AppSyncConfig.region,
  auth: {
    type: AppSyncConfig.authenticationType,
    apiKey: AppSyncConfig.apiKey,
  }
});

client.hydrated().then(function (client) {

  const observable = client.subscribe({ query: InboxSubscription, variables: { to: RECIPIENT } });

  const realtimeResults = function realtimeResults(data) {
    console.log('realtime data: ', data);
  };

  observable.subscribe({
    next: realtimeResults,
    complete: console.log,
    error: console.log,
  });
});

For this simple demo we have hardcoded the recipient to Bobby but obviously you’d want to make this dynamic for a real application.

8. Node.js Client Application Build Process

At this point we have all the source code written for our websockets client, we just need to implement the build process. Because the source code uses es6 features we need to transpile it using babel.

Install the dev dependencies we’ll need.

$ npm install --save-dev babel-cli babel-preset-es2015 rimraf webpack webpack-cli webpack-dev-server

Now let’s build our application.

$ rimraf build/ && babel ./src --out-dir build/ --ignore ./node_modules,./.babelrc,./package.json,./npm-debug.log --copy-file

9. Node.js Client Application

Now let’s run our application in node.js.

$ node build/index.js

Navigate to the queries page in your AppSync API in the AWS console and run the following graphQL mutation to trigger some udpates.

mutation Message {
    message(to: "Bobby", body: "Yo node!") {
        body
        to
        from
        sentAt
    }
}

You should see the message appear immediately in your terminal.

10. Browser Client Application Build Process

Remember the code we’ve written is isomorphic. That means it’ll run just as well in the browser.

First install the dev dependencies we’ll need.

$ npm install --save-dev webpack webpack-cli webpack-dev-server

We’re going to use webpack to run our build process so we need to create a config file for it. Create a file called webpack.config.js at the route of the project with the following contents.

const path = require('path');

module.exports = {
  entry: './src/index.js',
  output: {
    filename: 'bundle.js',
    path: path.resolve(__dirname, 'dist')
  }
};

As the webserver will be serving assets from the dist directory we need to add an index.html file that includes the bundle.js file which webpack will generate. Create a file at dist/index.html with the following contents.

<!DOCTYPE html>
<html>
<head>
  <title>AWS Serverless Websockets Demo</title>
</head>
<body>

</body>
<script type="text/javascript" src="bundle.js"></script>
</html>

10. Browser Client Application

Run the following command which will start webpack-dev-server serving assets from the dist directory.

$ webpack-dev-server --mode development --content-base dist/

Then navigate to http://localhost:8080 in your browser and open the dev tools as we’ll be logging the data to the console.

Navigate to the queries page in your AppSync API in the AWS console and run another graphQL mutation to trigger an update.

mutation Message {
    message(to: "Bobby", body: "hello browser!") {
        body
        to
        from
        sentAt
    }
}

You should see the updates being logged to your browser’s console.

12. Conclusion

We’ve set up a serverless GraphQL API with node.js and browser clients consuming realtime updates via websockets. Not bad for <30 minutes work!

Although AppSync is promoted as a managed GraphQL API service, for me its best feature is the ability to serve realtime updates. Previously you would have had to run a server 24/7 to do this. Now you get all the cost savings of serverless and without any of the headaches of managing servers.

Full source code for this serverless websockets example available on github.

If you’re interested in learning more about building realtime serverless applications then check out my upcoming training course Full Stack Serverless GraphQL Apps on AWS AppSync.

Build a GeoSearch GraphQL API using AWS AppSync & Elasticsearch

Thu, 07 Jun 2018 00:00:00 GMT

My guide to building a GeoSearch GraphQL API using AWS AppSync & Elasticsearch has just been published on the Serverless Framework blog.

In the tutorial you’ll learn how to build a GraphQL location search service similar to AirBnB’s using a fully serverless stack on AWS. Check out the full article over at the Serverless Framework blog.

If you’re interested in learning more about AWS AppSync then find out more about the training course I offer.

Why Startups Should Build on Serverless Platforms in 2018

Wed, 25 Apr 2018 00:00:00 GMT

When you go to startup communities like r/startups or indie hackers you often see non-technical founders asking for advice about which tech they should use to build their app. Invariably these end up in flame wars about development languages and frameworks. For me, this is totally missing the point. Yes ensure you are using experienced, talented developers. But the language or frameworks they use matters far less than the cloud platform you build your foundations on. That platform should be serverless.

What is serverless?

Serverless¹ is the umbrella term for the new breed of cloud services which do not require management of servers by customers. Hence server-less. It was first popularised by Amazon Web Services (AWS) in 2014 with the launch of their Lambda service. Since then AWS and other cloud providers have launched dozens more services, including databases, notifications and many more. The ecosystem has become so mature that large companies are now migrating their stacks wholesale to it.

Why is it such a big deal?

Because provisioning (configuring and installing the correct software) and managing (ensuring they remain active and secure) servers has historically accounted for a large percentage of the overall time it takes to develop and maintain a web application. Apart from the cost savings it brings, serverless also reduces the time to market for new products and features.

Only pay for what you use

Traditionally you would need to have a server running 24/7 to keep your application live. With serverless you are only charged when your application actually receives a request. With this pricing model you can keep your costs down to a minimum whilst you build up your customer base.

Scaling

Being able to deal with large numbers of users has been one of the hardest aspects of developing apps. With serverless, the cloud providers take on this responsibility and are able to do so on a massive scale, AWS claim that serverless is good for your “first ten million customers”.

The end of the protoype?

Traditionally startups would build a rough and ready first version which would need to be rebuilt from scratch as soon as they gained any traction. With serverless you no longer need to throw away your prototype when it becomes popular. If built correctly, it should scale to millions of users.

Go quicker, use serverless

In 2018, serverless is a mature technology with companies fast adopting it. It has made building an MVP quicker and cheaper than ever before. If you are a founder the first thing you should be telling your developers is “build on top of serverless platforms”.

If you enjoyed this article and want to read more like it then follow me on twitter or medium.

Confusingly there is also an application framework named serverless. It is just one of many tools you can use to develop on serverless platforms.
↩

How to Migrate a Go API to Serverless (in Under 10 Mins)

Tue, 17 Apr 2018 00:00:00 GMT

I’m going to step you through the process converting an existing Go API to serverless and deploying it to to AWS Lambda & API Gateway with AWS Severless Application Model (SAM). The whole process should take under 10 minutes. Let’s get started!

Set Up
Convert Application Code to Serverless
Build and Run Locally
Package Application for AWS Lambda
Deploy with AWS SAM
Test Deployed Serverless API

1. Setup

Our example API uses the HttpRouter package so let’s install that first.

$ go get github.com/julienschmidt/httprouter

$ tree
.
├── main.go
└── handlers.go

We have a single HTTP handler defined that will return a 200 HTTP response with the body ok.

# handlers.go
package main

import "net/http"

func HealthHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ok"))
}

Our entrypoint to the application, the main function, attaches the HealthHandler to the /healthz route and listens for HTTP requests on port 8080.

# main.go
package main

import (
    "fmt"
    "log"
    "net/http"

    "github.com/julienschmidt/httprouter"
)

const (
    serverPort = 8000
)

func main() {
    router := httprouter.New()
    router.Handler("GET", "/healthz", http.HandlerFunc(goserverlessapi.HealthHandler))

    fmt.Printf("Server listening on port: %d\n", serverPort)
    log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", serverPort), router), nil)
}

Let’s build and run this locally to check everything’s working okay.

$ go build -o go-serverless-api && ./go-serverless-api
Server listening on port: 8080

2. Convert Application Code to Serverless

In order to deploy to a serverless backend we need to be able to handle requests from AWS Lambda. Lambda functions have a different signature to regular HTTP handlers. Imagine if we had not one but hundreds in our application. We would have to manually update all the functions and re-write tests and in doing so we would forfeit our ability to deploy to non-serverless backends.

There is a solution which avoids all the problems just listed. We will create a modified entrypoint just for AWS Lambda. Using the gateway package we will swap out net/http’s ListenAndServe for gateway.ListenAndServe which will convert the payload that AWS Lambda provides into the *http.Request type that HTTP handlers accept.

In order to implement this second entrypoint we need to reorganise our project. We will create a directory for the AWS Lambda one and move our original entrypoint to a new folder too.

# original entrypoint moved to new location
$ mkdir -p cmd/go-serverless-api

# new entrypoint for lambda
$ mkdir -p cmd/go-serverless-api-lambda

We will copy the main.go file into each of these new directories and then remove it from the root of our project.

$ cp main.go cmd/go-serverless-api
$ cp main.go cmd/go-serverless-api-lambda
$ rm main.go

$ tree
.
├── cmd
│   ├── go-serverless-api
│   │   └── main.go
│   └── go-serverless-api-lambda
│       └── main.go
└── handlers.go

The package in the root of our project is no longer going to be the main package (the one Go uses to run your application). We will rename it to goserverlessapi so that we can import it as a library into our new entrypoints which will both become main packages.

$ grep -l 'package main' *.go | xargs sed -i 's/package main/package goserverlessapi/g'

After a simple find and replace on the Go files in the root of your project your handlers.go should look like this.

# handlers.go
package goserverlessapi

import "net/http"

func HealthHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ok"))
}

Now we need to update our original entrypoint to import the goserverlessapi package and access the HealthHandler function as an export from that. Note that you will need to modify the import path of the goserverlessapi package to match that of your project root’s location in your $GOPATH. The correct path for the example application on github used in this tutorial is github.com/techjacker/go-serverless-api.

# cmd/go-serverless-api/main.go
package main

import (
    "fmt"
    "log"
    "net/http"

    "github.com/julienschmidt/httprouter"
    "github.com/techjacker/go-serverless-api"
)

const (
    serverPort = 8080
)

func main() {
    router := httprouter.New()
    router.Handler("GET", "/healthz", http.HandlerFunc(goserverlessapi.HealthHandler))

    fmt.Printf("Server listening on port: %d\n", serverPort)
    log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", serverPort), router), nil)
}

Next update the AWS Lambda entrypoint. We are going to use the gateway package which is a drop-in replacement for Go net/http when running on AWS Lambda & API Gateway.

$ go get github.com/apex/gateway

# cmd/go-serverless-api-lambda/main.go
package main

import (
    "log"
    "net/http"

    "github.com/apex/gateway"
    "github.com/julienschmidt/httprouter"
    "github.com/techjacker/go-serverless-api"
)

func main() {
    router := httprouter.New()
    router.Handler("GET", "/healthz", http.HandlerFunc(goserverlessapi.HealthHandler))
    log.Fatal(gateway.ListenAndServe("", router), nil)
}

We have added gateway to our imports and swapped it out for http.ListenAndServe. The port value is redundant in the Lambda context and the gateway package discards it so we can safely remove the port constant and replace it with an empty string. In addition we have included our HealthHandler from the go-serverless-api package (the package in the root our of project which used to be our main package) as our handler for the /healthz path.

3. Build and Run Locally

Let’s build and run our original HTTP API again.

$ go build \
    -o go-serverless-api \
    ./cmd/go-serverless-api

$ ./go-serverless-api
Server listening on port: 8080

Open another terminal window and query it.

$ curl -s http://localhost:8080/healthz
ok

Everything still works!

Let’s do the same with the AWS Lambda version.

$ go build \
    -o go-serverless-api-lambda \
    ./cmd/go-serverless-api-lambda

$ ./go-serverless-api-lambda

Again, open a new terminal and query it.

$ curl -s http://localhost:8080/healthz
http: error: ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /healthz (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) while doing GET request to URL: http://localhost:8080/healthz

Don’t worry this error is expected as the AWS Lambda version is not listening for HTTP connections but instead expects to be fed an APIGatewayProxyRequest type.

4. Package Application For AWS Lambda

Build the binary for linux, the operating system that AWS Lambda uses.

$ GOOS=linux go build \
    -o go-serverless-api-lambda \
    ./cmd/go-serverless-api-lambda

AWS Lambda requires that function code be bundled into a zip so let’s go ahead and compress the binary.

$ zip go-serverless-api-lambda.zip go-serverless-api-lambda

We will be using the created go-serverless-api-lambda.zip in the final step - deployment.

5. Deploy

I’ve seen some tutorials that use the AWS CLI tools to deploy to AWS Lambda using ad hoc commands. This is absolutely the wrong approach to take! You should be automating your infrastructure just like every other aspect of your application. The industry standard for deployment is either Terraform or AWS Cloudformation. Both give you a declarative way to build your infrastructure. You save this configuration in YAML/JSON (Cloudformation) or HCL (Terraform) files which you commit to your repository. The problem with doing things this way is that you have to deal with all of the low level details of the stack. It would be nice if we had a way of describing our infrastructure at a high level in under 20 lines of code instead of hundreds. Enter AWS Severless Application Model (SAM).

AWS Severless Application Model (SAM)

SAM is a new standard spearheaded by Amazon aimed at making deploying serverless infrastructure simpler and more concise. SAM is an open source specification - see the full reference guide. Hopefully other cloud vendors will adopt this in the future and it will become possible to deploy seamlessly to multiple clouds with a single configuration.

Deploy with AWS SAM

Add the following YAML file to the root of your project. This is a SAM template that configures an AWS Lambda Function that runs your Go app and deploys it behind an HTTP interface provided by AWS API Gateway.

# template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: 'Boilerplate Go API.'
Resources:
  GoAPI:
    Type: AWS::Serverless::Function
    Properties:
      Handler: go-serverless-api-lambda
      Runtime: go1.x
      CodeUri: ./go-serverless-api-lambda.zip
      Events:
        Request:
          Type: Api
          Properties:
            Method: ANY
            Path: /{proxy+}

The line Type: AWS::Serverless::Function creates a Lambda function that is handled (Handler: go-serverless-api-lambda) by the binary file we built earlier. This Lambda function can respond to any number of events triggered by other AWS services such as those triggered by AWS S3 and Kinesis. The documentation contains the full list of event of sources. In our case we want it to respond to HTTP requests via API Gateway therefore we set the event to Type: Api. SAM implicitly creates an API Gateway for us as part of this which we then configure to respond to any type of HTTP request by setting the method to ANY). We tell our API to handle all paths below and including the root by adding Path: /{proxy+}.

We still need to upload the zip containing our Go binary to AWS S3. Ensure you have an S3 bucket created ready to receive our zip. This is a one-time operation you’ll want to do manually.

$ aws s3 mb s3://my-bucket

The following command will upload the zip and create a packaged SAM template.

$ aws cloudformation package \
    --template-file template.yaml \
    --s3-bucket my-bucket \
    --output-template-file packaged-template.yaml

You should now have a packaged-template.yaml file pointing to the uploaded zip.

# packaged-template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: 'Boilerplate Go API.'
Resources:
  GoAPI:
    Type: AWS::Serverless::Function
    Properties:
      Handler: go-serverless-api-lambda
      Runtime: go1.x
      CodeUri: s3://my-bucket/8982639e71e0d433cd99f9fa4207ecbe
      Events:
        Request:
          Type: Api
          Properties:
            Method: ANY
            Path: /{proxy+}

Now let’s deploy using this new packaged template.

$ aws cloudformation deploy \
    --template-file packaged-template.yaml \
    --stack-name go-serverless-api-stack \
    --capabilities CAPABILITY_IAM

The --capabilities CAPABILITY_IAM flag is required for cloudformation to create the stack for you as it will involve modifying IAM permissions - AWS mandate you set this explicitly as a safety measure. Under the hood the SAM template is compiled into a regular cloudformation template which is hundreds of lines longer. All of is completely hidden from the user (although you are free to inspect the compiled cloudformation template if you wish).

Setup 6. Test Deployed Serverless API

In order to discover the endpoint of our deployed API we need find out the API Gateway REST id.

$ aws apigateway get-rest-apis
{
    "items": [
        {
            "id": "0qu18x8pyd",
            "name": "go-serverless-api-stack",
            "createdDate": 1523987269,
            "version": "1.0",
            "apiKeySource": "HEADER",
            "endpointConfiguration": {
                "types": [
                    "EDGE"
                ]
            }
        }
    ]
}

AWS API Gateway addresses take the following format.

https://<api-rest-id>.execute-api.<your-aws-region>.amazonaws.com/<api-stage>

A stage is Amazon’s term for a deployment. SAM creates two different stages for you: Stage and Prod. Note the title case which is also used in the URLs! I think AWS forgot that everyone calls their test environment Staging not Stage but nevermind!

So SAM has set up endpoints for us at the following locations.

https://0qu18x8pyd.execute-api.eu-west-1.amazonaws.com/Stage
https://0qu18x8pyd.execute-api.eu-west-1.amazonaws.com/Prod

Let’s invoke our API.

$ curl -s https://0qu18x8pyd.execute-api.eu-west-1.amazonaws.com/Prod
{"message":"Missing Authentication Token"}

No need to panic! This is the standard API Gateway error when you make a request to the root resource without defining a handler for it. The only handler we have defined is /healthz, so let’s try that.

$ curl -s https://0qu18x8pyd.execute-api.eu-west-1.amazonaws.com/Prod/healthz
ok

Voila! Our API is now being powered by a serverless backend.

Full code for this tutorial available on github.

I’ll be posting more articles on Go and Serverless soon. Follow me on twitter or medium to get notified when I do.

Deploy your Flask API to any Serverless Cloud Platform using this Simple Pattern

Thu, 12 Apr 2018 00:00:00 GMT

I’m going to demonstrate how to make a Flask app serverless without locking you into any tooling or cloud platforms. It doesn’t require any application code changes or serverless frameworks so you can continue to develop locally like you always have.

Ideally you should re-architect your application into small discrete functions to take full advantage of serverless. However, if you want to migrate a legacy application or just a quick way to get started with serverless then this is a good approach to take.

Create Flask Application
Make the Application Serverless
Package the Application into Zip for Lambda
Deploy with Terraform

1. Create Flask Application

We are going to create a Flask application for Python3.6+.

Setup

First create a virtual environment and install dependencies.

$ python3 -m venv env
$ source env/bin/activate
$ pip install flask flask-lambda-python36 http-prompt

Application Code

Add the code below which will create a Flask Application with impeccable music taste.

$ tree -L 4
.
├── app
│   ├── api
│   │   ├── __init__.py
│   │   └── resources.py
│   └── __init__.py
└── run.py

# app/api/resources.py
from flask import Blueprint
from flask import request, jsonify

api = Blueprint('api', __name__)

@api.route('/artists', methods=('GET', 'POST'))
def handle_artists():
    if request.method == 'POST':
        return 'ok'

    return jsonify([{'name': 'enya'}])

# app/api/__init__.py
from .resources import api

# app/__init__.py
def create_app(Flask):
    app = Flask(__name__)
    from .api import api as api_blueprint
    app.register_blueprint(api_blueprint)

return app

# run.py
from flask import Flask
from app import create_app

http_server = create_app(Flask)

Complete application code available at the example repository.

Run the App

Run the app in local development mode.

$ FLASK_DEBUG=1 FLASK_APP=run.py flask run

Test the App

In another terminal window manually test the API.

$ http-prompt localhost:5000

http://localhost:5000> get artists

HTTP/1.0 200 OK
Content-Length: 31
Content-Type: application/json
Date: Thu, 12 Apr 2018 19:15:56 GMT
Server: Werkzeug/0.14.1 Python/3.6.5

[
    {
        "name": "enya"
    }
]

2. Make the Application Serverless

Up to this point this is just another Flask application. Time to make it serverless. Add another file called run-lambda.py to the root of the project.

# run-lambda.py
from flask_lambda import FlaskLambda
from app import create_app

http_server = create_app(FlaskLambda)

That’s it! Your application is now ready be run on AWS Lambda. The trick here is using FlaskLambda instead of normal Flask. FlaskLambda converts the payload that AWS feeds your Lambda function into the format that Flask expects.

This adaptor pattern is the same one that zappa uses. Personally I prefer not to introduce any extra tooling dependencies and instead opt for Terraform or AWS Serverless Application Model (SAM) for deployment.

Be aware that FlaskLambda only supports Python <= 3.5. I’ve created a fork that adds support for Python 3.6+ which you installed earlier with pip install flask-lambda-python36.

3. Package Application into Zip for Lambda

In order to deploy your application you need to package your application into a zip. The following script which will bundle your application code and dependencies.

# bundlelambda.sh
#!/usr/bin/env bash

OUTPUT_ZIP="$PWD/terraform/flask-app.zip"
ARCHIVE_TMP="/tmp/lambda-bundle-tmp.zip"

addToZip() {
    local exclude_packages="setuptools pip easy_install"
    zip -r9 s"$ARCHIVE_TMP" \
        --exclude ./*.pyc \
        --exclude "$exclude_packages" \
        -- "${@}"
}
addDependencies() {
    local packages_dir=()
    packages_dir+=($(python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"))
    env_packages=$(python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(plat_specific=1))")
    if [[ "$env_packages" != "${packages_dir[0]}" ]]; then
        packages_dir+=($env_packages)
    fi
    for (( i=0; i<${#packages_dir[@]}; i++ )); do
        [[ -d "${packages_dir[$i]}" ]] && cd "${packages_dir[$i]}" && addToZip -- *
    done
}

rm "$ARCHIVE_TMP" "$OUTPUT_ZIP"
addDependencies
addToZip app ./*.py
mv "$ARCHIVE_TMP" "$OUTPUT_ZIP"

This is a simplified version of the script in the example repository.

Save and run the script to create a zip bundle at terraform/flask-app.zip. Ensure you run this on a Linux platform to ensure compatibility with the Lambda runtime environment.

$ chmod +x bundlelambda.sh
$ ./bundlelambda.sh

4. Deploy with Terraform

Terraform Config

We will use the lambda-api-gateway terraform module to handle deployment. Create the Terraform configuration file at terraform/main.tf and update the account_id with your AWS account ID.

# terraform/main.tf
module "lambda_api_gateway" {
    source               = "git@github.com:techjacker/terraform-aws-lambda-api-gateway"

    # tags
    project              = "todo-mvc"
    service              = "acme-corp"
    owner                = "Roadrunner"
    costcenter           = "acme-abc"

    # vpc
    vpc_cidr             = "10.0.0.0/16"
    public_subnets_cidr  = ["10.0.1.0/24", "10.0.2.0/24"]
    private_subnets_cidr = ["10.0.3.0/24", "10.0.4.0/24"]
    nat_cidr             = ["10.0.5.0/24", "10.0.6.0/24"]
    igw_cidr             = "10.0.8.0/24"
    azs                  = ["eu-west-1a", "eu-west-1b"]

    # lambda
    lambda_zip_path      = "flask-app.zip"
    lambda_handler       = "run_lambda.http_server"
    lambda_runtime       = "python3.6"
    lambda_function_name = "HttpWebserver"

    # API gateway
    region               = "eu-west-1"
    account_id           = "123456789"
}

The key value here is lambda_handler which instructs AWS Lambda to use the run_lambda.py file as the entrypoint to our app.

Deploy

$ cd terraform
$ terraform apply

When complete, Terraform should output the API URL which you can use to manually test the deplopment succeeded.

$ http-prompt $(terraform output api_url)
GET artists

HTTP/1.0 200 OK
Content-Length: 31
Content-Type: application/json
Date: Thu, 12 Apr 2018 19:15:56 GMT
Server: Werkzeug/0.14.1 Python/3.6.5

[
    {
        "name": "enya"
    }
]

Conclusion

We took an existing Flask application and turned it serverless with just three lines of code! By using the adaptor pattern we have given ourselves the freedom to easily swap serverless cloud providers in the future. We could even switch to a container based deployment without having to change any of the application code.

Benefits of this approach:

Develop locally like you always have (no need for emulators)
Complete visibility into and control over your infrastructure
Avoids cloud vendor lock-in
Avoids application code framework lock-in

Future posts will cover AWS Serverless Application Model (SAM), Azure Functions and Google Cloud Function deployments.

Full code for this tutorial available on github.

How to Send Lightning Payments using the Terminal

Mon, 02 Apr 2018 00:00:00 GMT

In this guide I’m going to walk you through making a payment using Bitcoin’s Layer 2 Lightning Network using testnet to demonstrate. We will be using the command line as I’ve found this to be the most reliable way to use the network and doing so gives you a deeper understanding of how the network operates.

Prerequisites

You have access to a lightning node that: 1. is running bitcoind and lightningd and is synced to the testnet blockchain, 2. has blockstream’s c-lightning cli tool, lightning-cli, installed and accessible in its $PATH.

Check out my guide to setting up a lightning network node if you still need to set one up.

1. Generate Address and Fund Wallet

Layer 1 BTC transactions = 1. Wait Time = 60 mins.

Create a new address.

$ lightning-cli newaddr

Go to the testnet faucet and send yourself some bitcoin. Tip - use chrome to get that 70% bonus for using IPV6!

Click on the transaction ID link that will appear just beneath which will take you to the blockchain explorer with details of the transaction.

You need to wait for 6 confirmations - about 1 hour before the funds show up in your wallet.

$ lightning-cli listfunds
{
	"outputs": [
		{
			"txid": "d30ad72742d5c05343f65a1f78a90191f7b13341ed309cad0297a4e2fd3f0196",
			"output": 1,
			"value": 64849668,
			"status": "confirmed"
		}
	]
}

2. Connect to a Peer and Create a Payment Channel

Layer 1 BTC transactions = 1. Wait Time = 60 mins.

In order to make payments on the network you first need to connect to it. How do you find another node to connect to? Head to Acinq’s lightning network explorer.

Search for a well connected peer (one with a lot of channels) and copy its URI. I timed out on a few nodes initially, just keep trying and you’ll find one that connects immediately.

$ lightning-cli connect 03e5f9d1935c67a029bf0a26af5f63109f4f4c45993a8f45177a8f2f39bcb46144 77.58.162.148 9735

$ lightning-cli list peers
		{
			"state": "GOSSIPING",
			"id": "03e5f9d1935c67a029bf0a26af5f63109f4f4c45993a8f45177a8f2f39bcb46144",
			"alias": "IslandDSP",
			"netaddr": [
				"84.246.200.122:9735"
			],
			"connected": true
		}

Now we will create a payment channel with the peer we just connected to. Opening and closing channels entails interacting with the main blockchain so we want to fund it with a decent amount (in millisatoshis) so that we can use it for several payments.

$ lightning-cli fundchannel 03e5f9d1935c67a029bf0a26af5f63109f4f4c45993a8f45177a8f2f39bcb46144 50000

$ lightning-cli listpeers
{
			"id": "03e5f9d1935c67a029bf0a26af5f63109f4f4c45993a8f45177a8f2f39bcb46144",
			"connected": true,
			"netaddr": [
				"84.246.200.122:9735"
			],
			"alias": "IslandDSP",
			"color": "68f442",
			"channels": [
				{
					"state": "CHANNELD_AWAITING_LOCKIN",
					"owner": "lightning_channeld",
					"channel_id": "958b3bd04adb40fd8ff0700a0b24e0298a2ac265273c08ed0218af38140ba678",
					"funding_txid": "78a60b1438af1802ed083c2765c22a8a29e0240b0a70f08ffd40db4ad03b8b95",
					"msatoshi_to_us": 500000000,
					"msatoshi_total": 500000000,
					"dust_limit_satoshis": 546,
					"max_htlc_value_in_flight_msat": 18446744073709552000,
					"channel_reserve_satoshis": 0,
					"htlc_minimum_msat": 0,
					"their_to_self_delay": 6,
					"our_to_self_delay": 144,
					"to_self_delay": 6,
					"max_accepted_htlcs": 483,
					"status": [
						"CHANNELD_AWAITING_LOCKIN:Funding needs more confirmations."
					],

Again, search for the txid in the bitcoin testnet explorer and wait for 3-6 confirmations before the channel is considered open. When it is you should see the status changed to CHANNELD_NORMAL.

$ lightning-cli listpeers
{
	"peers": [
		{
			"id": "03e5f9d1935c67a029bf0a26af5f63109f4f4c45993a8f45177a8f2f39bcb46144",
			"connected": true,
			"netaddr": [
				"84.246.200.122:9735"
			],
			"alias": "IslandDSP",
			"color": "3399ff",
			"channels": [
				{
					"state": "CHANNELD_NORMAL",
					"owner": "lightning_channeld",
					"short_channel_id": "1289768:416:0",
					"channel_id": "ba95c71228ce3f57e25f6ac9c4a7418a52a056435b2276dd47437c7521ebda8e",
					"funding_txid": "8edaeb21757c4347dd76225b4356a0528a41a7c4c96a5fe2573fce2812c795ba",
					"msatoshi_to_us": 100000000,
					"msatoshi_total": 100000000,
					"dust_limit_satoshis": 546,
					"max_htlc_value_in_flight_msat": 18446744073709552000,
					"channel_reserve_satoshis": 0,
					"htlc_minimum_msat": 0,
					"their_to_self_delay": 6,
					"our_to_self_delay": 144,
					"to_self_delay": 6,
					"max_accepted_htlcs": 483,
					"status": [
						"CHANNELD_NORMAL:Funding transaction locked. Channel announced."
					],
					"in_payments_offered": 0,
					"in_msatoshi_offered": 0,
					"in_payments_fulfilled": 0,
					"in_msatoshi_fulfilled": 0,
					"out_payments_offered": 0,
					"out_msatoshi_offered": 0,
					"out_payments_fulfilled": 0,
					"out_msatoshi_fulfilled": 0
				}
			]
		}
	]
}

3. Generate Payment Request

Now we have a confirmed channel established we can start using the lightning network for transactions.

Go to an article on Y’alls and click on the “continue reading” button to generate a payment request. This is essentially a serialised invoice. Decode it to check that you are not being overcharged!

$ lightning-cli decodepay lntb1500n1pdvp7t9pp5k5mvp8u045pdnumnt89cdc7l90lpvff7sjzkrvawg93ju3mgg3nsdp62fjkzepqg9e8g6trd3jn5gzfyp9kummhyptksmeq2dshgmmndp5jqjtnp5cqzyskw8lzg7w4hcfyhncczcaexlpx3tdefm0vjnh9dkqev23g94tdta8jjqzp87v6r9q8xwe3mnjf0tryaa7zkxxj4gwgfcqea32j0x069qqsjyjhz
{
	"currency": "tb",
	"timestamp": 1522596197,
	"created_at": 1522596197,
	"expiry": 3600,
	"payee": "039cc950286a8fa99218283d1adc2456e0d5e81be558da77dd6e85ba9a1fff5ad3",
	"msatoshi": 150000,
	"description": "Read Article: I Know Who Satoshi Is\r",
	"min_final_cltv_expiry": 144,
	"payment_hash": "b536c09f8fad02d9f37359cb86e3df2bfe16253e848561b3ae41632e47684467",
	"signature": "3045022100b38ff123ceadf0925e78c0b1dc9be13456dca76f64a772b6c0cb151416ab6afa022079480209fccd0ca0399d98ee724bd63277be158c69550e42700cf62a93ccfd14"
}

4. Make the Payment

Layer 2 BTC transactions = 1. Wait Time = milliseconds.

Now we have everything in place to make a layer 2 bitcoin transaction which should be instantly confirmed.

$ lightning-cli pay lntb1500n1pdvp7t9pp5k5mvp8u045pdnumnt89cdc7l90lpvff7sjzkrvawg93ju3mgg3nsdp62fjkzepqg9e8g6trd3jn5gzfyp9kummhyptksmeq2dshgmmndp5jqjtnp5cqzyskw8lzg7w4hcfyhncczcaexlpx3tdefm0vjnh9dkqev23g94tdta8jjqzp87v6r9q8xwe3mnjf0tryaa7zkxxj4gwgfcqea32j0x069qqsjyjhz
{
	"code": 206,
	"message": "Fee 1100 is 0.733333% of payment 150000; max fee requested is 0.500000%",
	"data": {
		"fee": 1100,
		"feepercent": 0.733333,
		"msatoshi": 150000,
		"maxfeepercent": 0.500000,
		"getroute_tries": 6,
		"sendpay_tries": 0,
		"failures": []
	}
}

Go back to Y’alls and click on the “check for payment” button to reveal the article. Unfortunately this didn’t work for me. When I checked on the command line I discovered that indeed no payments had been made.

$ lightning-cli listpayments
{
	"payments": []
}

To try to solve this issue, I attempted to create a payment channel with Y’alls directly (their address is listed as “peer” when they create you an invoice to continue reading).

$ lightning-cli connect 039cc950286a8fa99218283d1adc2456e0d5e81be558da77dd6e85ba9a1fff5ad3 34.200.252.146

$ lightning-cli fundchannel 039cc950286a8fa99218283d1adc2456e0d5e81be558da77dd6e85ba9a1fff5ad3 500000
{
	"code": -1,
	"message": "received ERROR channel 3795216ba4a257a1f36f1062af58b4513ad82dac90d298e45ca83271e67cf1d6: Unacceptable channel constraints: minimum HTLC value is too large: 0"
}

Apparently this is a known bug with lnd, which Y’alls must be using as their lightning daemon.

As a backup plan we’ll send a payment to ourselves using a web lightning wallet. Go to htlc.me and generate a wallet and then click on the “receive” tab to request a payment of 0.0001 tBTC ($0.65 tUSD). Copy the serialised invoice and check it before sending payment.

$ lightning-cli decodepay lntb100u1pdvzzc4pp5h55942ehdpy5g70sty9mnvt3k2wlvt4ucnctnellau8drhaew8lsdqqcqzyswv3n3vyzx9uua4tjl8ttcp5fp7srn80zywvg4vses6fj4q9n3v846mgkuu648u37venf93znkckadjejvddv04vyq5l2krx486z23ysqhcrxg6
{
	"currency": "tb",
	"timestamp": 1522600725,
	"created_at": 1522600725,
	"expiry": 3600,
	"payee": "03193d512b010997885b232ecd6b300917e5288de8785d6d9f619a8952728c78e8",
	"msatoshi": 10000000,
	"description": "",
	"min_final_cltv_expiry": 144,
	"payment_hash": "bd285aab3768494479f0590bb9b171b29df62ebcc4f0b9e7ffef0ed1dfb971ff",
	"signature": "30440220732338b0823179ced572f9d6bc06890fa0399de223988ab21986932a80b38b0f02205d6d16e73553f23e666692c453b62dd6cb32635ac7d584053eab0cd53e84a892"
}

This time when I attempted payment I received multiple errors.

$ lightning-cli pay lntb100u1pdvzzc4pp5h55942ehdpy5g70sty9mnvt3k2wlvt4ucnctnellau8drhaew8lsdqqcqzyswv3n3vyzx9uua4tjl8ttcp5fp7srn80zywvg4vses6fj4q9n3v846mgkuu648u37venf93znkckadjejvddv04vyq5l2krx486z23ysqhcrxg6
{
	"failures": [{
			"type": "FAIL_PAYMENT_REPLY",
			"erring_index": 1,
			"failcode": 4103...

However, when I refreshed the htlc.me web page I saw that the payment had in fact succeeded. Checking in the console confirmed that payment had been made.

$ lightning-cli listpayments
{
	"payments": [
		{
			"id": 1,
			"payment_hash": "bd285aab3768494479f0590bb9b171b29df62ebcc4f0b9e7ffef0ed1dfb971ff",
			"destination": "03193d512b010997885b232ecd6b300917e5288de8785d6d9f619a8952728c78e8",
			"msatoshi": 10000000,
			"msatoshi_sent": 10002020,
			"timestamp": 1522601027,
			"created_at": 1522601027,
			"status": "complete",
			"payment_preimage": "3e75637befbeec9e7056a4103a06d652cd3cdee9134de69799d8b39c6b1f714c"
		}
	]
}

We sent a payment and paid just 2020 millisatoshi in transaction fees ($0.00013 USD at the time of writing)!

Finally, let’s try to connect with htlc.me directly so we don’t get any console errors when making payment. Copy the payee id from the decodepay command output above and search for it in the lightning network explorer. Copy the URI then connect to it and create a channel.

$ lightning-cli connect 03193d512b010997885b232ecd6b300917e5288de8785d6d9f619a8952728c78e8 34.201.11.250 9735

$ lightning-cli fundchannel 03193d512b010997885b232ecd6b300917e5288de8785d6d9f619a8952728c78e8 500000

Unfortunately we need to wait for the transaction to confirm on the blockchain so that the channel becomes usable.

An hour later we can see the channel is established - the status is CHANNELD_NORMAL.

$ lightning-cli listpeers
	{
		"id": "03193d512b010997885b232ecd6b300917e5288de8785d6d9f619a8952728c78e8",
		"connected": true,
		"netaddr": [
			"34.201.11.250:9735"
		],
		"alias": "htlc.me",
		"color": "854cc5",
		"channels": [
			{
				"state": "CHANNELD_NORMAL",
				"owner": "lightning_channeld",
				"short_channel_id": "1289949:56:0",
				"channel_id": "20dd321c23224b2022f62f59602efc311a8ad63c4481fcd210ef8dade88e8064",
				"funding_txid": "64808ee8ad8def10d2fc81443cd68a1a31fc2e60592ff622204b22231c32dd20",
				"msatoshi_to_us": 500000000,
				"msatoshi_total": 500000000,
				"dust_limit_satoshis": 546,
				"max_htlc_value_in_flight_msat": 18446744073709552000,
				"channel_reserve_satoshis": 0,
				"htlc_minimum_msat": 0,
				"their_to_self_delay": 6,
				"our_to_self_delay": 144,
				"to_self_delay": 6,
				"max_accepted_htlcs": 483,
				"status": [
					"CHANNELD_NORMAL:Funding transaction locked."
				],
				"in_payments_offered": 0,
				"in_msatoshi_offered": 0,
				"in_payments_fulfilled": 0,
				"in_msatoshi_fulfilled": 0,
				"out_payments_offered": 0,
				"out_msatoshi_offered": 0,
				"out_payments_fulfilled": 0,
				"out_msatoshi_fulfilled": 0
			}
		]
	}

Now generate another invoice on htlc.me, this time for 0.00020000 tBTC ($1.31 tUSD).

$ lightning-cli decodepay lntb200u1pdvzvf8pp5tyjfw9ma9uj8hhrhqvmum9vzvaun45qg6mmesu3c08zkhmgl44ssdqqcqzysah4qe4zef3rssusa42cyx0gxxq6q06g50m38w7chz50en80dfh45fj7kdgpx474udehq22mv2pwxfky66kdjdrw88z34w8hjjdjdu6sq6zy2a4

Finally make the payment.

$ lightning-cli pay lntb200u1pdvzvf8pp5tyjfw9ma9uj8hhrhqvmum9vzvaun45qg6mmesu3c08zkhmgl44ssdqqcqzysah4qe4zef3rssusa42cyx0gxxq6q06g50m38w7chz50en80dfh45fj7kdgpx474udehq22mv2pwxfky66kdjdrw88z34w8hjjdjdu6sq6zy2a4
{
	"id": 2,
	"payment_hash": "592497177d2f247bdc770337cd958267793ad008d6f798723879c56bed1fad61",
	"destination": "03193d512b010997885b232ecd6b300917e5288de8785d6d9f619a8952728c78e8",
	"msatoshi": 20000000,
	"msatoshi_sent": 20000000,
	"timestamp": 1522610489,
	"created_at": 1522610489,
	"status": "complete",
	"payment_preimage": "1e2767675a227c7f8d57252bfe67d6324829c267f3df38396b4d2bb79a27dfd6",
	"getroute_tries": 1,
	"sendpay_tries": 1,
	"route": [{
		"id": "03193d512b010997885b232ecd6b300917e5288de8785d6d9f619a8952728c78e8",
		"channel": "1289949:56:0",
		"msatoshi": 20000000,
		"delay": 144
	}],
	"failures": []
}

This time no console errors. Your web wallet at htlc.me should immediately detect the payment. You should now be able to make any number of subsequent payments instantly and with similarly low fees until you deplete the funds in your channel or it expires.

Conclusion

As you can see, the nodes on testnet are sketchy to say the least. The three main lightning software implementations (lnd, c-lightning, eclair) are all still in the early stages of development so it’s not surprising that a few bugs exist. But this should at least give you a flavour of how cheap and fast transactions will be in the future when the network is bigger and the software running it more mature.

Join the discussion on medium.

How to Set Up a Lightning Network Node (4 Easy Steps)

Fri, 30 Mar 2018 00:00:00 GMT

This guide will take you through setting up a machine running ubuntu/debian as a lightning network node.

1. Install Bitcoind

$ sudo add-apt-repository -y ppa:bitcoin/bitcoin
$ sudo apt-get update
$ sudo apt-get install -y bitcoind

2. Install C-Lightning

c-lightning is blockstream’s lightning network daemon.

$ sudo apt-get install -y \
	autoconf automake build-essential git libtool libgmp-dev \
	libsqlite3-dev python python3 net-tools
$ cd /opt
$ git clone https://github.com/ElementsProject/lightning.git lightningd
$ cd lightningd
$ make

3. Configure Bitcoind and C-Lightning

Create a file at ~/.lightning/config with the following contents.

# network=testnet # uncomment to run on testnet instead of mainnet

Create a file at ~/.bitcoin/bitcoin.conf with the following contents.

# testnet=1 # uncomment to run on testnet instead of mainnet
printtoconsole=1
server=1
txindex=1

4. Sync with the blockchain

This step will take 1-4 hours depending upon the speed of your internet connection.

Run the bitcoin daemon.

$ bitcoind -daemon

In another terminal window run the lightning daemon.

$ lightningd

Check everything is running correctly.

$ lightning-cli -H getinfo
{
	"id": "07798d536c58ede3d3a150ba89679e2dc78a3a0966074024ad4081367abda76719",
	"port": 9735,
	"address": [
		{
			"type": "ipv6",
			"address": "edc7:c9f:dc33:5922:21b2:f0d0:2a02:4f40",
			"port": 9735
		}
	],
	"version": "v0.5.2-2016-11-21-2438-g1fe79df",
	"blockheight": 1290015,
	"network": "testnet"
}

Check the blockheight of your target blockchain using an explorer to discover how far through the syncing process you are.

Conclusion

You’ll want to be running your node on machine that is permanently switched on and connected to the internet. Restarting the daemons even after they’ve downloaded the blockchain takes a long time as they have to reindex transactions in memory. Furthermore, you’ll want your node constantly available so it can accept and route payments, earning you BTC.

Personally I’ve used an old laptop which I leave switched on and connected to my router 24/7. For convenience I’ve set up ssh access to it so I can make payments using it. Check out my guide on how to make lightning payments to learn how to do this.