Client

MeU’s client is a US-based company that has been around for 15 years in the recruitment industry. The company’s product is a job searching engine that serves thousands of job posts in hundreds of categories and locations.

Challenges

• • • • • In need of a large scale web application development – Job Board Network under the main brand and saturate SERPs with local and niche search term domination and domain authority
        • Efficiency indexing and full texting searching on hundred thousands of jobs
        • The company owns about 3000 different domains for local and niche search purposes. All these domains must have the same front-end codebase which interfaces with users.
        • The admin system must allow deploy new or take down a local site in a matter of minutes and all parameters must be configurable.
        • Building a high scalability system based on a network of AWS services
        • Automate the deployment process to maintain a different stage of development and testing.

Solutions

• • • • • The development team decided to build a Job Crawler Engine which running in the background and managed by the supervisor to aggregate job posts from big job engines like Indeed, Monster, SimplyHired
        • The development team decided to index the job posts using the ElasticSearch engine. The match queries with fuzziness in Elastic help search millions of job fast and reliable
        • The system was build based on API-based architecture. So basically, we separate the front-end which interface with the user include CSS, JS, HTML page, rendering the views, etc into a delicate code-base and deploy to app servers, the back-end code which working with database, external APIs, background processing, etc will be put into another code-base and run on API servers.
        • Building the system with a high-scale mindset from the beginning. The team builds a system architecture spitted web server and data server, along with HAProxy load-balancing in place. The code designed to work with multiple data servers and multiple web servers.
        • Employed Chef into the dev-ops, plus the Jenkin build system. We also have different configuration-based environments for development, staging, and production. The team can deploy the major build into production or staging in a matter of minutes without any manual works.

Results

• • • • • The team with 5 developers has been completed this system from design till production in 90 days
        • With well-built Elastic Search schema and indexing, the search around million of jobs results in fast and reliable. Along with lazy load on the design, the search requests mostly under seconds.
        • The API-based architecture takes the development team a lot of extra time, in the beginning, to follow and adding extra work hours into programming. However, it really pays off. The system very robust and maximum scalability. Also, the well-organized architecture separates the API server, and the web app system increase productivity in the end, easier to maintain, and debug when the system goes live. Actually, the system has been live for more than 1 year but we didn’t see any major downtime or any critical bugs happened.
        • The development team has successfully built a local site deployment tool that allows deploying any new niche job site that serves a niche job market that provides the same UX as the primary site in a matter of minutes with all configuration being one on the admin panel. The whole process of deploy new code to a new server, set up the configuration, put the site into Live mode, or maintenance mode all being automated.
        • The system has good scalability. We can increase or reduce resources such as EC2, Amazon RDS, etc depend on a load of the system in a matter of hours, and all can be done via AWS command.
        • Automate the headache process of maintaining mode, deploy new build and testing new build in staging system before pushing into production, etc.