Job Board Network - Large Scale Web Application Development Case Study
Client is a US based company which have been around for 15 years in recruitment industry. The company’s product is a job searching engine which serves thousands of job posts in hundreds of categories and locations.
The project need to solve following major problems:
- In need of a large scale web application development-Job Board Network under the main brand and saturate SERPs with local and niche search term domination and domain authority
- Efficiency indexing and full texting searching on hundred thousands of jobs
- The company owns about 3000 difference domains for local and niche search purpose. All these domains must have same front-end codebase which interface with users
- The admin system must allow deploy new or take down a local site in matter of minutes and all parameters must be configurable.
- Building a high scalability system based on network of AWS services
- Automate the deployment process to maintain difference stage of development and testing.
- Development team decided to build a Job Crawler Engine which running in background and managed by supervisor to aggregate job posts from big job engines like Indeed, Monster, SimplyHired
- The development team decided to indexing the job posts using ElasticSearch engine. The match queries with fuzziness in Elastic help search millions of job fast and reliable
- The system build based on the API-based architecture. So basically, we separate the front-end which interface with user include CSS, JS, HTML page, rendering the views etc into a delicate code-base and deploy to app servers, the back-end code which working with database, external APIs, background processing etc will be put into a another code-base and run on API servers.
- Building the system with high-scale mindset from beginning. The team builds a system architecture spitted web server and data server, along with HAProxy load-balancing in place. The code designed to work with multiple data servers and multiple web servers.
- Employed Chef into the dev-ops, plus the Jenkin build system. We also have difference configuration-based environment for development, stagging and production. Team can deploy the major build into production or staging in matter of minutes without any manual works.
- The team with 5 developers has been completed this system from design till production in 90 days
- With well-built ElasticSearch schema and indexing, the search around million of jobs results in fast and reliable. Along with lazy load on the design, the search requests mostly under seconds.
- The API-based architecture takes the development team a lot of extra time at beginning to follow and adding extra work hours into programming. However, it really pays off. The system very robust and maximum the scalability. Also the well-organized architecture separate the API server and web app system increase the productivity at the end, easier to maintain and debug when system go live. Actually, system has been live for more than 1 year but we didn’t see any major down-time or any critical bugs happened.
- The development team has successfully build a local site deployment tool which allow deploy any new niche job site which server a niche job market which provide same UX as the primary site in matter of minutes with all configuration being one on the admin panel. The whole process of deploy new code to new server, setup the configuration, put site into Live mode or maintain mode all being automated.
- System has good scalability. We can increase or reduce resources such as EC2, Amazon RDS etc depend on the load of system in matter of hours, and all can be done via aws command.
- Automate the headache process of maintain mode, deploy new build and testing new build in staging system before pushing into production etc.