Skip to content

Jesses Software Engineering Blog

Sep 03

Jesse

AWS DynamoDB Setup

DynamoDB is AWS’s NoSQL solution. While not the most robust NoSQL solution, DynamoDB offers low latency at scale and is a fully managed service. There are some features that are a bit odd, like pre-defining table throughput and a per index pay structure, but they’re relatively minor hurdles when compared to DynamoDB’s scalability and cost. Like other services, DynamoDB is pay per use, and is a great tool for prototyping new products as cost will be close to zero with light usage.

Local Setup

AWS offers a downloadable version of DynamoDB for local development. This allows for both query writing and throughput/access pattern analysis, and only requires Java.

java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb

After starting the .jar file locally (make sure the path correctly points to the .jar), DynamoDB can either be accessed via command line

aws dynamodb list-tables --endpoint-url http://localhost:8000

a web browser for interactive querying, which is a powerful tool that allows inline querying with JavaScript and offers a complete library of common DynamoDB commands


http://localhost:8000/shell

or programmatically using an AWS SDK and the localhost endpoint. NOTE: The region is required when using the SDKs.

const dynamodb = new AWS.DynamoDB({
  apiVersion: '2012-08-10',
  endpoint: 'http://localhost:8000',
  region: 'us-west-2' // required
});

There are also a few configurations to customize the local database instance.

Basic Querying

DynamoDB’s query language is straightforward and simple. DynamoDB is essentially a key value store with some basic querying capabilities and compound indexing. DynamoDB has all the functions expected, such as reading/writing data (optionally as batch), creating/modifying tables, and adding indexes. One thing to note is there is a DocumentClient for interacting with the DynamoDB documents, which is simpler than the SDKs. According to the SDK documentation, to pull docs based on key a type is needed:

{
 TableName : 'Company', 
 Key : {
    id : { 
      N : 12314 
    } 
  },
 AttributesToGet:  [ 'id', 'name']
}

The query is specifying that the id field is a number. But with the DocumentClient, the type can be dropped, making for a simpler query block. The DocumentClient provides a variety of simpler querying structures.

{
    TableName: 'Company',
    Key: { 
        id: 12314, 
    },
    AttributesToGet: [ 'id', 'name']
};
Simple Queries

# LIST TABLES

var params = {
    ExclusiveStartTableName: 'table_name',
    Limit: 0, 
};

dynamodb.listTables({}, () => (e, d) {});

# CREATE TABLE – DynamoDB is “schemaless” allowing only index definitions. Also consider how items should be stored.

var params = {
    TableName: 'entity',
    KeySchema: [ 
        { 
            AttributeName: '$id',
            KeyType: 'HASH'
        }
    ],
    AttributeDefinitions: [ 
        {
            AttributeName: '$id',
            AttributeType: 'N' 
        },
    ],
    ProvisionedThroughput: { 
        ReadCapacityUnits: 1, 
        WriteCapacityUnits: 1, 
    }
};

dynamodb.createTable(params, () => (e, d) {});

Alternatively, a global index can be set up. NOTE: The index has it’s own provisioning as that’s what is queried. The projection value specifies which columns to place into the query. Alternatively a LocalSecondaryIndexes can be set up; however, keep in mind it must be a compound key also containing the primary index’s HASH value. Also, the local keys do not have separate provisioning, they use the same throughput definitions as the parent table.

var params = {
  TableName: 'entity',
  KeySchema: [
    {
      AttributeName: 'DocId',
      KeyType: 'HASH',
    },
    {
      AttributeName: 'EntryTime',
      KeyType: 'RANGE',
    },
  ],
  GlobalSecondaryIndexes: [
    {
      IndexName: 'entity-source',
      KeySchema: [
        {
          AttributeName: 'SourceId',
          KeyType: 'HASH'
        }
      ],
      Projection: {
        ProjectionType: 'ALL'
      },
      ProvisionedThroughput: {
        ReadCapacityUnits: 1,
        WriteCapacityUnits: 100,
      }
    }
  ],
    AttributeDefinitions: [
    {
      AttributeName: 'DocId',
      AttributeType: 'S',
    },
    {
      AttributeName: 'EntryTime',
      AttributeType: 'N',
    },
    {
      AttributeName: 'SourceId',
      AttributeType: 'S',
    },
  ],
  ProvisionedThroughput: {
    ReadCapacityUnits: 1,
    WriteCapacityUnits: 1,
  },
};

# PUT – Note the use of the DocumentClient and using put() not putItem()

var params = {
    TableName: entity,
    Item: { 
        $id: 12314,
        ename: 'jesse'
    }
};

docClient.put(params, () => (e, d) {});

# GET – The get() will only retrieve docs based on the indexes. Can also use the query() call to specify ranges or conditions but the query() call must include a partition key.

var params = {
    TableName: 'entity',
    Key: { 
        $id:12314, 
    },
    AttributesToGet: [ '$id, name]
};

docClient.get(params, () => (e, d) {});

# QUERY

// pull by HASH
var params = {
  TableName: 'entity',
  KeyConditionExpression: 'DocId = :id',
  ExpressionAttributeValues: {
    ':id': '87a869e976c18882fjid890de81eb1b6'
  }
};

// filter on a field
var params = {
  TableName: 'entity',
  KeyConditionExpression: 'DocId = :id AND EntryTime >= :entrytime',
  ExpressionAttributeValues: {
    ':id': '87a869e976c18882fjid890de81eb1b6',
    ':entrytime': 1479081600000,
    ':field': 'value',
  },
  FilterExpression: 'field = :field',
};

// filter with BETWEEN
var params = {
  TableName: 'entity',
  KeyConditionExpression: 'DocId = :id AND EntryTime BETWEEN :from AND :to',
  ExpressionAttributeValues: {
    ':id': '87a869e976c18882fjid890de81eb1b6',
    ':from': 1479081600000,
    ':to': 1479081600000
  },

};

# SCAN – Scan will scan the entire table. The FilterExpression is applied prior to returning the data set. A maximum of 1MB can be returned. Scans can be ran in parallel for better performance.

var params = {
  TableName : 'entity',
  FilterExpression : 'ename = :name',
  ExpressionAttributeValues : {':name' : 'jesse'},
  ProjectionExpression: '$id',
};

docClient.scan(params, () => (e, d) {});

When building the tables and indexes in AWS, there is a UI to help set up the schemas. The best practices documentation provides important information on setting up a scalable DynamoDB architecture. For more in-depth querying overview, see the full documentation.

Blog Powered By Wordpress