AllShare TV – Media Server

Now you can turn your iPhone or iPad into a Media Server and share your photos and videos with Smart TV. If you have a DLNA compatible Media Playback Device (e.g. TV, Playstation, X-Box, Blu-Ray Player etc…) you can flip your videos and photos from your iOS device to your TV Screen without connecting any wires.

You can install AllShare TV application on your iPhone / iPad and it will discover compatible media rendering devices (e.g. Smart TV) in your home network. You need to connect your iOS device and Smart TV to same Wifi router and these devices will seamlessly integrate with each other. Now you select a video or photo from your iOS device to play on TV or you can use the TV remote to browse your photos and videos on iPhone / iPad.

This application implements the DLNA protocol so you can use it with various media products including Samsung Smart TV, Panasonic Viera, Sony Bravia Internet TV, Sony PS3, Microsoft X-Box, LG Smart TV. However you need to make sure that your TV supports DLNA technology.

For more information, checkout the detailed feature list in AppStore d

Download from AppStore

Redis — NoSQL Data Store

Overview

As mentioned in my previous post “Next Generation Data Storage“, that NoSQL is not a silver bullet. It’s all about the right tool for the task as compared to all-purpose generic RDBMS. So the key here is to select the right data-store for your needs and developing good understanding of features provided by that data-store. In this post I will try to explore some wonderful set of features provided by Redis data store and will discuss some use case scenarios where Redis can make your life easier. Redis is an open-source, in-memory, persistent key/value data-store sponsored by VMWare. It is also referred to as the data-structure store as the keys can hold different data-structures as their value. Redis supports master-slave replication such that the data from any Redis server can be replicated to any number of slaves. Redis data store is written in ANSI C and it works on most POSIX systems like Linux, Mac OS X, FreeBSD, Solaris. The official release for Windows is not available but there are windows ports available using MinGW.

When to use Redis?

Although it depends a lot on the detailed requirements of you application but there are some scenarios when you should consider the option of using Redis data store. Consider the following situations,

• Your data small enough to be contained in RAM
• You need super high-performance key/value data-store
• You can afford to lose some recent updates in case of system crash

In case the answer is yes then there is a fair chance that Redis can make your life simpler. Due to the fact that Redis is an in-memory store, it cannot hold data more than the available space in your RAM. You can configure Redis to use virtual memory but still all the keys must fit in RAM and besides that it is not a brilliant option to use Redis with virtual memory option. This limitation enables the Redis to provide super-fast performance as the data is directly served from the RAM.

Memory is the new disk. Disk is the new tape.
–Jim Grey

Redis provides two different approaches to persist the data depending upon your use case. One is semi-persistent durability mode achieved through snapshots where data from the memory is asynchronously dumped to the disk periodically, and the other one is the safer approach where each command is appended to the log. So if the your data can fit in memory and you can afford asynchronous writes then Redis can be a good choice.

The Redis Advantage

Now the most important part; How to take full advantage of Redis in your application. Before discussing some interesting ways to use Redis for common problems, I want to repeat two important points that I discussed before,

1. It’s totally fair to use multiple data-stores in same project, so you can use some NoSQL data-store for some portion of your project and use different data-store or RDBMS for some other portion of your same project.
2. When designing your application’s data model using a NoSQL data-store, you have to think different as compared to conventional RDBMS.

The best thing that I like about Redis is that you can offload lot of responsibility to your data-store. Redis is a data-structure storage system which is lot more than just a dumb data-store. You don’t need to follow the conventional way by fetching the data from database in your application and populating your data-structures before you process the data based on your algorithms. In Redis you store the data in form of different data-structures and you can perform some interesting operations on data. Following data-structures are supported by Redis,

• Strings
  Support simple GET, SET, DEL operations.

  SET  key  value     /* Store the specified values for given Key */
  GET  key            /* Retrieves the value for given Key */
  DEL  key            /* Delete the given key */

• Lists
  Support variety of List operations using simple commands,

  LPUSH myList 1      /* Prepend value '1' to list named myList */
  LLEN  myList        /* Get the length of list */
  LPOP  myList        /* Remove and get first element of the list */
  RPOP  myList        /* Remove and get last element of the list */
  LTRIM myList 0  1    /* Trim the list from index 0 to index 1 */
  LRANGE myList 0  2   /* Get all the values from index 0 to index 2*/

• Sets
  Support many useful operations for SET manipulation,
  

  SADD set value /* Adds value to the SET represented by "set" */
  SREM set value /* Remove value from the specified SET */
  SMEMBERS set /* List all the members of specified SET */
  SISMEMBER set value /* Check if set contains the specified value */
  SUNION set1 set2 /* Take union of both sets */
  SINTER set1 set2 /* Take intersection of both sets */
  SDIFF set1 set2 /* Subtract the set2 members from set1 */

• Sorted Sets
  Support many useful operations on sorted SET,

  ZADD set 10 value    /* Adds value to the SET with score 10 */
  ZRANGE set 0 2    /* Get values in sorted SET between index 0 and 2 */
  ZRANGEBYSCORE set 10 30   /* Get values in sorted SET with score 10 to 30 */ 
  ZINCRBY set 10 value    /* Increment score of value by 10 */

  Mostly retrieval functions are available with for operations on reverse sorted set. The functions start with ZREV… (e.g. ZREVRANGE etc..)

• Hashes
  Support various functions for hash store manipulation,

  HSET key field value    /* Set value for field in hash store named "key" */
  HGET key field        /* Get value for field in hash store named "key" */
  SMSET key f1 v1 f2 v2 /*Set multiple fields of hash store represented by key*/
  HMGET key f1 f2 f3    /* Get multiple fields (f1,f2 and f3) from "key" */
  HKEYS key    /* List all fields in hash store represented by key */

• Atomic increments and Expiration
  Support useful features like Atomic increments and expirations,
  

  INCR key    /* Increment integer value of key by 1 atomically */
  DECR key    /* Decrement integer value of key by 1 atomically */
  EXPIRE key 5    /* Expire key automatically after 5 seconds */
  TTL key        /* Get time to expire for specified key */

Let’s consider a Web-2.0 solution with social networking features like Facebook and see how Redis compares to conventional RDBMS.

Display recent messages

To start, we consider a very common scenario where you have to display the 10 recent messages on user’s homepage.

 1: /*Push the message id of new messages to the list*/

 2: LPUSH user:msgs {msg-id}

 3:

 4: /*Get latest message ids (0 to 10) from the list*/

 5: LRANGE user:msgs 0 10

 6:

 7: /*To get next 10 message ids from (10 to 20)*/

 8: LRANGE user:msgs 10 20

You can limit maximum number of messages for a user to 500 by adding a LTRIM after every LPUSH like,

 1: LPUSH user:msgs {msg-id}

 2: LTRIM user:msgs 0 500

This will ensure that a user’s inbox will have the latest 500 messages only.

Display Counters

Let’s say you want to display the counters on your website like Votes, Page Hits, File Downloads etc… You can use Atomic increment operations whenever your page is accessed or a file download is requested like,

 1: INCR counters:downloads:{file-id}

 2: INCR counters:downloads:{today-date}:{file-id}

 3: INCR counters:hits:{page-id}

If you counter already exists, it will be incremented else a new counter will be created and incremented. Please note that 2nd statement in above code will manage a file download counter representing the downloads every day separately. To display these counters in your application you can just get the value of any counter like,

 1: GET counters:downloads:{file-id}

 2: GET counters:downloads:{today-date}:{file-id}

 3: GET counters:downloads:{page-id}

Connections and Memberships

Let’s see what happens when a user adds a connection (Add a friend) or joins some group in your social network application. You can use SETS to manage this scenario by creating a SET for user’s connection (e.g: “users:{user-id}:connections”) and a SET for group members (e.g: “groups:{group-id}:members”) and whenever user joins a group you can add the user-id to the SET like,

 1: SADD users:{user-id}:connections   {user-id-of-connection}

 2: SADD groups:{group-id}:members  {user-id}

Now when user visits a group page and you need following information,

• List of group members
• Is user a member of this group?
• All the connections of user who also joined this group

You can get this information by simple operations like,

 1: SMEMBERS groups:{group-id}:members

 2: SISMEMBER groups:{group-id}:members  {user-id}

 3: SINTER users:{user-id}:connections   groups:{group-id}:members

In above code example, we are using “SMEMBERS” command to list all the members of SET in the 1st line, and in 2nd line we are using “SISMEMBER” command to check if the user-id exists in given SET of group members. In the 3rd line of code we are taking the intersection of two sets (i.e. group members and user’s connections) to find the values that exist in both sets (Users that exist in current user’s connections and also the group members).

Now let’s say we import new contacts from email address book and create a new SET (e.g: users:{user-id}:contacts) for all these contacts. Assuming that we are using “user emails” as user-id in the SETS, here are few more operations

Display contacts that are not added as connections,

 1: /* Take Difference of two SETS */

 2: SDIFF users:{user-id}:contacts   users:{user-id}:connections

Display everyone including email contacts and connections in this social network,

 1: /* Take Union of two SETS */

 2: SUNION users:{user-id}:contacts  users:{user-id}:connections

Currently Nearby connections

Let’s extend the previously created SETS to display the connections or friends that maybe nearby (Feature like Google-Check-in or Facebook-Places ). For the purpose we will maintain some new SETS for registered places and we will display the connections that checked-in at some place nearby user in around 5 – 10 minutes. Now Consider the following example,

Whenever some user Check-In at some place we will add that user to the SET of that place,

 1: /*Roy checks-in to Fight Club*/

 2: SADD checkIn:FightClub:fresh  roy@ttl.com

 3:

 4: /*Ken checks-in to Fight Club*/

 5: SADD checkIn:FightClub:fresh  ken@ttl.com

After every 5 minutes we move the fresh set to the stale set to purge old check-ins and also we will store the union of fresh check-ins and stale check-ins to a new SET “checkIn:FightClub”. Please note that moving/renaming the fresh SET to stale SET will overwrite the existing contents of stale SET.

 1: RENAME checkIn:FightClub:fresh checkIn:FightClub:stale

 2: SUNIONSTORE checkIn:FightClub  checkin:FightClub:fresh checkIn:fightClub:stale

Now when a new user check-in to same place (i.e: FightClub), that user can check to see if some connection or friend is nearby,

 1: /*Now Blanka enters the Fight Club and checks if someone is nearby*/

 2: SINTER users:blanka@ttl.com:connections  checkIn:FightClub

The above code statement will give the Blanka’s network connections that are also member of Fight Club. And as we are refreshing the contents of Fight Club by rotating fresh and stale sets, so a user is automatically removed from Fight Club after 5 – 10 minutes of check-in.

Leaderboard Management

Another quick use can be a game application in your social network. When lot of people are playing the game and registering their scores, you may require to handle hundreds of Add and Update requests for user scores. For cases like that you can use Sorted Sets in Redis for a high performance implementation. Consider the following simple operations,

 1: /* User A got 10 points */

 2: ZINCRBY scores 10 A

 3: /* User B got 15 points */

 4: ZINCRBY scores 15 B

 5: /* User A got another 10 points */

 6: ZINCRBY scores 10 A

 7:

 8: /* Display top 10 scores */

 9: ZREVRANGE scores 0 9 WITHSCORES

 10:

 11: /* Display the rank of User A */

 12: ZRANK scores A

 13:

 14: /* Display the score of User A */

 15: ZSCORE scores A

Publish/Subscribe Pattern

Redis also provides high-performance implementation of Publish/Subscribe pattern which you can access this functionality using very simple commands like SUBSCRIBE, UNSUBSCRIBE, PUBLISH.

The Publish/Subscribe implementation also provide pattern-matching subscriptions. You can subscribe to a channel using “ PSUBSCRIBE log.* ”. This will enable you to receive event whenever a message is published matching “log.*” pattern (e.g: “PUBLISH log.Info” or “PUBLISH log.error”).

You can do a lot more with Redis like there is are blocking variants of List POP command in Redis that blocks if the list is empty, so you can use Redis Lists as Queues. Similarly you can use Redis Sorted Sets to implement Priority Queues.

Footnotes:

1. http://redis.io/
   
   

Next Generation Data Storage

Overview

When we say database, many people considers RDBMS as their only option and of course its been years that people are successfully developing solutions powered by relational databases. There was a time not long ago when it seemed like that a 3-tier architecture is a silver bullet to serve the needs of data centric solutions. However the the rapidly changing trends call for the solutions that can scale out to large number of hosts and more intelligent architectures like SOA and Cloud. As far as this article is concerned, I am not going to talk much about SOA and cloud computing but it worth mentioning that these architectures placed new challenges for the database and it was too much to swallow for the RDBMS. The relational databases were not meant for scaling out to hundreds of machines on a dispersed network so stretching these databases didn’t solved the problems and resulted in increased costs to maintain expensive hardware and increased complexity.

To meet the challenges NoSQL databases gained fame and in no time there are dozens of NoSQL databases in the market and lot more in the making. Some of the biggest organizations replaced their relational databases with specialized NoSQL databases. Google, Facebook, Yahoo and other organizations with need to handle huge data are now using NoSQL databases. But NoSQL is not just for big data organizations, in fact it is gaining it popularity in new startups and small to medium Web-2.0 companies.

The name ‘NoSQL’?

Before going further, I just want to share my thoughts about the term “NoSQL”. The name NoSQL is not about “No SQL”. Of course NoSQL databases does not provide SQL interface but the main difference of NoSQL databases is their non-relational and schema free nature so it would be more precise to call them schema free databases.

Schema free storage

The NoSQL systems do not enforce schema on the database which brings lot of advantages and new possibilities. For example if you store record of ‘Users’ in schema free database then it is not necessary that all the users have same set of fields. If some user do not have “Cell-Number” in profile and other users have this field then it is not necessary to have this ‘Nullable’ filed for all the users like in RDBMS. Similarly you can add new fields on the fly without any impact on your design. Also I find it very very simple to migrate the data from a NoSQL database to a new storage structure as I just have to define the mappings and there are no schema constraints to care about.

Not a silver bullet !!

Now the most important thing that once should know before using the NoSQL data storage system for the next project. NoSQL is not a silver bullet and no matter they give wonderful set of features and unmatched performance, it is not feasible to go for NoSQL whenever you need to store some data. In my point of view the main motivation behind the invention of NoSQL databases is to create a specialized data storage system for specialized needs instead of using a generic RDBMS for all scenarios. So it is mandatory to understand the use-case scenarios of  different NoSQL databases before thinking about using it. Sometimes it is better to go with RDBMS depending upon the scenario and nature of your project.

NoSQL data model

In contrast to rational model used by RDBMS, there are different data models used by different NoSQL databases. Following are the commonly used models,

• Key based storage models
• Key / Value Stores
• Key / Data-structure stores
• Key / Document stores
• Column Family Sotres
• Graph Storage models

Key / Value Stores

The simplest of all are the Key/Value storage models and these data storage systems are as simple as they sound. This type of databases supports GET and SET functionality that can be used to place or retrieve the data against a Key. In this class of data storage systems Riak and Voldemort are popular examples. Voldemort provides a distributed data store that is used by LinkedIn (Famous Professional Network Website).

Key / Data-structure stores

The Key/Data-structure stores (one of my favorites) are highly useful data storage systems. Instead of storing the dumb data you can store and manipulate different data-structures in these data storage system. You can store data against a Key in form of Queues, Lists, Sets etc… which helps you offloading lot of code and mess from your application. Redis is the most popular data-structure storage system that provides unmatched set of features and performance. If you want to start playing with NoSQL, then Redis is a MUST TRY!!!

Key / Document stores

Key/Document store maps a key to a document that contains the structured information. This class of NoSQL databases provide rich set of features and you can query or filter records as the data is stored in structured documents. Some of very popular database systems in this category are MongoDB, CouchDB and RavenDB. I personally see very bright future for Key/Document based databases, however using these databases in your application is relatively complex and requires some strong considerations regarding organization of your data.

Column Family Sotres

The column family based databases are inspired by the Google’s BigTable  which is the proprietary database used by Google (Not available in the market). These databases are designed to scale over thousands of machines and can handle huge data distributed across these machines. Cassandra, Hbase and Hypertable are famous examples of such databases. In this model, a key identifies a row, which contains data stored in one or more Column Families. Each row can contain multiple columns within a column family. The values within each column are timestamped, so that several versions of a row-column mapping can live within a column family. These are the databases that powers the Facebook, Yahoo, Google and other big data giants.

Graph Storage models

The Graph storage databases stores the data in form of Graph data structure, which is a fundamental data structure in computer science. The HyperGraphDB and Neo4J are two popular databases for storing graph structured data. The graph storage is very different from previously mentioned data storage techniques in every aspect including, storage architecture, data traversal, querying the data, transactional semantics and also distribution pattern of data across multiple machines. The graph storage may not be a very good choice for storing all type of data but for some scenarios its makes the life easier if you store and query data in garph structured storage systems.

Conclusion

The NoSQL databases can be a very useful choice for data storage if you know how and which model to use. The NoSQL is still in its infancy and evolving and there is lot to come in this domain. Unlike RDBMS, different NoSQL storage systems provide different approaches to handle the data and give a lot of command to application designer to play around. So it is very important that one should understand different storage models and their use cases to squeeze the juice out of these wonderful storage systems.

This article was just a rough overview and I will be writing more on the topic with focus on design and development using specific NoSQL databases.

Advice

I know many people are already on board but still want to write a piece of advice for some designers and architects who are reluctant to experiment (as per my observation).

It is not necessary to use a single storage model for your whole solution. Its totally acceptable and a good practice to use different storage models for different tasks in the same solution until you are using the right tool for the job. Try shifting your application logging to MongoDB leaving rest of your application to use the old RDBMS; its a good thing, Not a sin.

Footnotes:

1. http://redis.io
2. http://www.mongodb.org
3. http://couchdb.apache.org
4. http://ravendb.net
5. http://project-voldemort.com
6. http://www.hypergraphdb.org
7. http://www.basho.com
8. http://neo4j.org
9. http://labs.google.com/papers/bigtable.html
10. http://hadoop.apache.org
11. http://hbase.apache.org
12. http://cassandra.apache.org