SEEKAsia Engineering team

Having 200+ engineers and a 15+ years of legacy softwares from Jobstreet and JobsDB, we certainly had to tweak our processes to coup with the increasing scaling issues.

We had to solve a problem :

Scaling and performance issues with application processing.

We had a single objective:

Application Indexing & Search with ElasticSearch

And some key results to achieve:

  • Able to scale 10x applications from current
  • Able to adapt changes fast - re-index < 3 days
  • Acceptable performance even scale to 10x

Here are some of the things we think we learned/did well while solving above problem.

A/B testing ( Split testing )

One of the important parts of the project was to maintain the current process and not having any downtime, as well as gaining confidence from the stakeholders on the technical design or approach we are taking.

A/B testing or Split testing helped us to build confidence in our solution. We could monitor for anomalies as well as test and do a performance comparison between old and new systems.

This also helped us to do tests on incremental changes and performance iterations fast without impacting the full system.

Autonomous Team

Teams of employees who typically perform highly related or interdependent jobs, who are identified and identifiable as a social unit in an organization, and who are given significant authority and responsibility for many aspects of their work, such as planning, scheduling, assigning tasks to members, and making decisions with economic consequences. – Guzzo & Dickson

Let’s talk about our autonomy:

External autonomy

We had the project stakeholders to steer us to the right direction. But they also gave us the total autonomy in terms of decision-making, technology adaptation, system design and flexible delivery timings. They gave us the different perspective on certain problems which we got via the constant feedback loop that we had via our conversation driven development.

Internal Autonomy

As a team we were always discussing possible solutions, sense of ownership and freedom to try out technologies which gave our team the total control of each of the part we are developing or re-developing or retiring. We constantly debated our solutions and tried improving on the given ideas which gave us the drive to achieve better things.

Individual autonomy

As a team member, we had certain decision authority as well as the sense of ownership and collaboration. Pro-actively monitoring for bugs, performance enhancements re-structuring certain part of the code came from this.

Traditional leadership roles became redundant because of the autonomy.

Development Lifecycle

We wrote code mainly in two parts. Infrastructure and API. API development lifecycle was faster when we started doing production releases via hotfixes. We still did development via develop branch when we needed major tests on some of our APIs in staging or QA environment.

We started as kanban; mixed extreme programming in between but broke a dozen of ( maybe less ) extreme programming principles.

What we took from extreme programming:

  • Main activities ( Coding, Testing, Listening & Designing )
  • Short development cycle ( Short iterations )
  • Embracing changes
  • Feedback loop

What we broke of Extreme Programming principles:

  • We were not convinced on the idea of solutions will be simple.
  • Pair programming (didn’t have many of those; we fall back to code review)
  • Test Driven Development ( TDD )

This is a diagram of micro deployments done by appindex team in a nutshell:

Development Lifecycle

Here is a more in-depth read up on micro deployments in uber’s engineering blog.

In the second part of the journey, I will talk about Feedback loop, Monitoring, Communication, Collaborative decision making, Performance enhancements iterations, debates/arguments and most importantly how we learned to fail fast.