Client

Meu’s client is a start-up company with USD 6M funding. The company’s product helps creative professionals around the world share and get inspiration for their creative works. The social network site grew to 60K professional and 300K high-quality artworks.

Challenges

• • • • • Process a high-quality media which up to 20MB/photo, also video and audio. The system has to doing background processing on the uploaded media to make it load faster on browser
        • Efficiently query social activities to display on the user’s dashboard who’s following up to 5000 creative. The traditional relational database such as MySQL has a problem with the limitation in implementation.
        • Build a recommendation system based on user’s activities and favorites
        • Build a high scalability system based on a network of AWS services
        • Automate the deployment process to maintain different stages of development and testing.
        • Mobile applications with limit memory, the network must load and render media efficiently and smoothly

Solutions

• • • • • Build a queued jobs based on beanstalkd, a Memcached-based system. Its interface is generic but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously. The job system monitored by a supervisor, a process control system. The supervisor monitored and spawn the background process if its exited or killed by OS for any reason. When media being uploaded into the system, a new job created and pushed into the queue and triggered a process to generate thumb, resize, video processing, etc using the FFmpeg and uploading to AWS CloudFront. This system increased the responsive time of the system for heavy jobs significantly. We also apply this system to other jobs example sending the email, generate weekly reports, scheduled clean system, etc
        • The team had a difficult time with modeling and querying the network data using MySQL. Later, we decide to go to another database, a graph-based database system called neo4j. The primary database still MySQL, however all the social follower networks, social activities being modeling on the graph-database.
        • The mobile app utilizes the SD card to cache image and frees the cache periodically to make it efficient using the space to improve the loading experience but also not consume too much space on SD cards. The mobile app dashboard can scroll up to 200 images without any lagging.
        • Build the system with a high-scale mindset from the beginning. The team builds a system architecture spitted web server and data server, along with HAProxy load-balancing in place. The code designed to work with multiple data servers and multiple web servers.
        • Employed Chef into the dev-ops, plus the Jenkins build system. We also have different configuration-based environments for development, staging, and production. The team can deploy the major build into production or staging in a matter of minutes without any manual works.

Results

• • • • • Maintain multiple sizes of media, which serve different media quality based on the user’s browser, also the background system removes blocking users from continuing their activities on the website. The loading time reduces significantly from 30s to minutes to under 5s.
        • With minify system, CDN and the optimized on HTML templates, YSlow Performance score from E to B
        • Query activities from 4000 network users down from minutes in MySQL to seconds with graph-database. The further optimization reduces the load from 7s to under 2s, and low depend on network size
        • The system has good scalability. We can increase or reduce resources such as EC2, Amazon RDS, etc depend on a load of the system in a matter of hours, and all can be done via AWS command
        • Automate the headache process of maintaining mode, deploy new build and testing new