In today's article, I will discuss scaling your engineering organisation. It is an easy task to hire developer's and get them coding on new projects, and to continue hiring like crazy as more people are needed. However, it can get to a particular stage where this doesn't keep making sense. The teams all start to reinvent the wheel too many times over, everyone starts creating their own repo's, tooling, front end frameworks and style guides and it all blows up when it hits QA, and they struggle to test everything. This article won't focus on any departmental structures, but more around the piece's of the puzzle you need to have in place to scale effectively. Let us get started.

What is scaling?

When I talk about scaling your engineering Organisation, I am referring to increasing the size and capabilities of your engineering team(s). Your company may start out with developing a single product for their customer's. Over time, the product may require a web and mobile front end or may require a globally scaled cloud deployment or require security certification to enter a regulated market. The company may create new products that require all of these elements also.

Typically when significant new requirements come into play, we need additional people to assist in getting the work done. Now it's easy enough to go out and hire people like crazy, give them all a laptop and wish them good luck. However, this group of people won't magically organise themselves into an efficient engineering organisation.

Scaling refers to the act of building up your engineering function most efficiently and effectively to accommodate these new requirements from the business. All businesses want to make the best use of their resources and effectively scaling your engineering function is a way to achieve this.

Problems with scaling

Businesses can run into various issues when scaling an engineering function. The issues below cause a drain on resources and reduce the efficiency of the engineering function, not to mention increasing its cost base. I have listed a few of these below.

No shared vision or strategy

One of the significant issues we face with engineering organisations at scale is the lack of a strategic vision. Starting with an overall product strategy, but also extending to development, technology and architecture strategy. Without a shared vision/strategy, the different engineering squads evolve in separate directions, making independent decisions on the areas mentioned above: Product, Technology, architecture and overall development strategy. Rewinding these decisions down the road is a costly exercise, which can leave people disgruntled and disengaged.

Reinventing the wheel

A big issue encountered when scaling engineering organisations is the almost universal reinventing of the wheel. It must be human nature, but unless teams are (strongly) encouraged to share and re-use elements, re-use does not occur. Which can be costly for the organisation as effort is being duplicated many times over, with the re-use issue only getting worse with increased scale. With the various engineering teams spending time reinventing the wheel, they are unable to spend time on delivering business value or innovation.

Teams do their own software thing

Each team is comprised of individuals with varied experiences, these people bring their own experiences with developer tooling, software packages and approaches to development. Before too long, we have many different packages in use across the organisation. Which is ok if your engineering function is small consisting of one or two teams; however, if you have 50 different teams, this becomes an issue. We lose the ability to standardise and develop expertise in a particular package which costs the company in terms of efficiency and effectiveness. Also, we lose any benefits around volume pricing the company may have obtained.

Expertise cannot grow.

When we have a large development organisation split into many small squads or teams working hard to deliver their software, expertise is often tough to develop. People don't have time to become a wizard in Architecture, Test automation, DevOps, UX, Technical writing all because they have a task to do that requires a little bit of these skills but not the time to invest in order to become a star. The parent company in question is still spending the same amount of money but across all the different business units instead of just one, creating a decentralised inefficiency which doesn't delivery business value.

Teams don't work well together.

Another common issue that occurs when there are multiple engineering teams in one organisation is friction between the teams. Which often occurs in a competitive culture, where teams compete with against each other. In contrast to a culture where everyone is on the same team, where it is possible for teams to be successful without some other team losing. This type of culture can rot an engineering organisation and stifle innovation and delivery of the business products.

Key ingredients for effective scaling?

The below items are what I consider to be key ingredients for scaling an engineering organisation.

Effective engineering leadership, strategy and direction

Effective leadership of your engineering organisation is one of the critical elements required to be able to scale effectively. Effective leadership covering critical areas of engineering practice enables important decisions to be debated and made about product, engineering, architecture and people strategy. People need to understand the strategy and direction that the business is heading down; it provides a framework and guiding beacon for their efforts. Leadership is not a one-horse race; we are talking about a group of people who bring their experience and expertise to bear to lead the engineering organisation.

Scaling requires investment

To properly scale any engineering organisation investment is required to build out the core services and expertise required to obtain scale. The amount of investment you will require is related to the size and speed of the scaling you require. Payback periods will also be affected by these decisions.

An inclusive team-based culture

One of the first tasks the leadership should undertake is to cultivate an inclusive team-based culture. It is essential that people feel part of the team striving for a common goal. As the team(s) grows, it is essential to ensure that a collaborative spirit that runs through the organisation. It is essential for people to feel that someone doesn't have to lose for them to win.

Roles and pathways are clearly defined

Like any organisation, people need to understand what their contribution is to the overall goals of the business, which is painted in broad strokes by the strategy piece, but given more detail and focus in the position/role descriptions. The organisation needs to have a cohesive set of position descriptions which detail what people do and how their effectiveness is measured.

Expertise is cultivated and grown

People have the opportunity to specialise and grow their expertise in a specific area via groups called centre's of excellence. These centres of excellence develop elite-level skills in critical areas such as architecture and design, automation and DevOps, UI/UX, et al. to be leveraged organisation-wide.

Automate where possible

Engineering teams at scale automate as much as possible. Testing, deployments, data functions are all places where repetitive tasks can be automated to enable an engineering team to scale. Automation needs to be a guiding principle for the engineering organisation.

Maximise re-use

Another guiding principle that needs to be adopted is to maximise re-use across the organisation. Re-use of code libraries, UI/UX patterns, frameworks, et al., should be a guiding principle and encouraged in the organisation. The ability to stand on the shoulders of those people that came before them will enable those folks to spend more of their time innovating or delivering business value.

How to scale your engineering organisation?

In the below list, I have detailed some recommendations for scaling an engineering organisation, which can be implemented at any time of your engineering organisation lifecycle; however, it is most cost-effective to implement these before attaining scale. This list is not exhaustive but will provide a good foundation for scaling.

Leadership Structure

A strong leadership structure is essential for a scaled engineering organisation, with the correct focus on the technical area's critical for scaling. Apart from the head of the engineering role that has the ultimate responsibility for engineering department the following senior roles/responsibilities are required to set a foundation of core services that the development squads (teams) can leverage.

• Architecture & Design
• Application Security
• Design (UI/UX)
• Quality
• DevOps/Infrastructure
• Documentation and training
• Support
• Analysis and process

These senior leaders will be responsible for setting up practices in these respective areas that provide frameworks and specialist services in these areas to the development squads.

Engineering squads

Development activities take place in engineering squads. The development squads are self-contained teams (Dev lead, Developers, Business Analysts, Testers). Squads are created, merged and altered as needed to cater to the resourcing needs.

Engineering Managers

Engineering managers are responsible for the delivery of a stream. Engineering managers have several squads reporting into them and require a delivery manager to assist with the coordination of development activities and leveraging shared services (DevOps, test automation, design) where required. As the organisation grows, additional engineering managers can help manage the up-tick in squads. In a scaled organisation, engineering managers focus on their particular value stream they are delivering, unlike engineering managers in smaller companies that may have a broader role to fill.

Product strategy

Product strategy sits typically outside the engineering organisation; however, there is the need for a close relationship with product strategy to ensure we have captured the user's voice. Also, product strategy should ensure that product roadmaps are created and shared to ensure the entire organisation is aware of the product direction. Another key area of that product strategy should work hard to leverage is the re-use of existing products, modules into new products with small changes to cater to a different market or vertical.

People

People are the main ingredient of your engineering organisation. Several areas require focus to enable scaling.

Career pathways

The organisation needs to identify, document and use career pathways in the management of their people. The worse thing for innovation is when people stagnate in a particular role for many years, maintaining the status quo. Ideally, career pathways are defined for all major disciplines, providing a range of choice for folks to find something that can drive them forward in their career.

Position descriptions

PD's spell out the requirements and responsibilities for a particular role. It is essential to standardise PD's for performance measurement across the organisation, in addition to aligning them to the career pathways. Thereby ensuring everyone is on the same pages in regards to what's expected of them and how they progress throughout the company. These two tools enable the organisation to manage staff performance and career progression effectively.

Standardised hiring process

To make hiring as simple as possible, the process of hiring should be standardised and provided as a service to the engineering managers. Leveraging standard position descriptions and career pathways, hiring should be a straightforward process.

Communities of practice

Another great way to build capability and share information, expertise and advice in an engineering organisation is to establish communities of practice. Communities of practice are a forum where groups of like practitioners can gather regularly to share ideas, know-how and best practice. Communities of practice cover each of the key development areas including coding, business analysis, quality, UX, DevOps and many more. Communities of practice need to be established and driven by the senior leadership for each discipline to ensure that best practices are adopted, information shared, and the organisation achieves re-use of assets.

Core systems

The engineering organisation needs to select one set of core systems. Core systems in development terms include:

• Source code repositories
• IDE's
• Development management systems
• Knowledge management
• Cloud infrastructure Standardising on a set of core platforms enables the organisation to save costs with volume licensing and enable a shared knowledge and expertise to be developed. The communities of practice and the organisation's centres of excellence can be involved in reviewing and selecting the core systems in use, regularly to ensure that the business is using the best systems.

Centre's of Excellence

Centre's of excellence or shared services are teams of people focussed on a particular area of expertise. The idea of a centre of excellence is that in-depth knowledge and skills developed within a centre of excellence can then be provided as a service to the rest of the organisation. Examples include:

• Architecture & Design: Responsible for the design, architecture, coding standards for the software under development, ensuring consistent use of technologies and architectural patterns. The Architecture and Design Centre of Excellence (COE) will promote re-use of code components/modules/libraries/APIs by maintaining and promoting the organisations shared code repositories to reduce rework by developers.
• Application Security: This group sets the SSDLC (Secure software development life cycle) and standards within, to ensure that all development is developed with a security lens applied. This groups consulting services apply to security space, including penetration testing and DevSecOps consulting.
• Design (UI/UX): This group sets design guidelines for the entire engineering group, ensuring a consistent set of user interfaces and design artefacts (HTML, CSS, images, et al.) within the development squads. This group provides consulting services around user journeys, user testing and design.
• DevOps/Infrastructure: This group is responsible for setting the foundations for the engineering organisation's automation, build, infrastructure/cloud automation and deployment efforts. This group will consult the organisation around DevOps and cloud infrastructure automation and requirements.
• Documentation and Training: Are responsible for setting the direction in terms of creating documentation and training artefacts. This group will own documentation repositories and any client-facing documentation and training needs.
• Support: This group handles all application support needs for the business. They have a client-first focus and development resources to make code fixes where needed.
• Analysis and process: This team focuses on the development process and (business) analyst domain. They will advise teams on the best practice development process and provide specialist resources such as coaching, scrum masters, et al. The centre of excellence model can go a long way to ensuring re-use across the vital domain areas of the engineering team. Centralising these services will prevent the squads from needing to reinvent the wheel in these critical areas. Another benefit of the centre of excellence model is innovation, which can happen in a few ways. Firstly the centre of excellence model allows the regular development squads to focus more on the business requirements at hand, and not worry around the specialised tasks the centre of excellence can provide, thereby freeing up time to focus on value-added innovation. Secondly, the centre of excellence themselves can innovate with their singular focus on their domain of expertise.

Re-use the holy grail of software

Another critical area of importance when scaling an engineering organisation is to ensure that crucial concepts, artefacts, libraries, API's, frameworks, design system are all only created once and re-used many times over. Centre's of excellence have a big part to play here, sharing the critical outputs of their teams with the broader organisation. Re-use needs to be a mantra of the engineering organisation.

Technologies

Where possible the technology stack should be kept as streamlined as possible. There are always use cases where the tech stack can diverge, however it is a benefit for the organisation to keep a streamlined consistency, which will aid re-use and up-skilling within the organisation.

Handovers between team's

A significant issue in some companies is defining how teams work together. In the model discussed in this article, there are many points of interoperation between teams that occur to make everything work properly. If we do not define proper handover processes between teams, it can lead to inefficient and inconsistent handover practices, not to mention constant bickering between the teams. I feel that putting these handovers into a systematic process (i.e. Jira or Azure DevOps et al.) is the way to go. This way, the system enforces any rules and is reproducible across the organisation.

Culture

A key element for the successful scaling of an engineering organisation is to ensure an inclusive and healthy culture permeates throughout. The culture needs to reflect a meritocracy in whereby efforts and achievements, as opposed to politics, plays a significant role in success. The senior leadership team needs to work hard to ensure that a safe environment where people are empowered to speak up and challenge when they see things they believe are wrong is the norm.

No timesheets

The final point I will make around scaling an engineering organisation is that timesheets should be avoided at all costs. Timesheets are inaccurate and a waste of precious engineering resources which should be avoided at all costs. With proper development planning and careful application of development methodologies, any organisation should be able to divine how much money they are spending on the engineering and even down to the product line level if the above model is implemented. Just don't do it.

Conclusion

In the above article, I have presented a few considerations for the scaling of an engineering organisation. As every company is different, so too will the approaches are taken to scale their engineering department. I hope some of the considerations I have put forward will be of use in scaling your engineering organisation.