At face-value it is always going
to be more cost-effective to perform labour, irrespective of the industry, in
countries where the labour cost is cheaper. It’s a fact that is hard to argue
against, almost impossible really, and it has been happening for years in a
large number of industries such as manufacturing and banking. If you can charge
out a resource at a third of the price of an onshore resource then it is a compelling
proposition not just for you, but for your customers as well. With respect to IT,
offshoring managed service work, packaged software implementations, helpdesk
and call centre operations (the kind of work that is highly governed by prescriptive
processes and is accompanied with numerous repeatable, measurable steps i.e. no
need to “think” so much as to just “do”) it usually will be. That’s the proposal
being sold by companies that can offshore and you really you can’t argue with
it. But when that same line is used for bespoke software development the
situation is different. Very different.
Bespoke application development
requires a number of different set of skills and qualities – that aren’t just
confined to an organisations or teams competencies with a given set of
technologies required to deliver a solution. It requires you to think
laterally, logically, and in entirety with respect to the solution being built.
You need to be able to think outside the box to get things done, either as a
developer, tester or architect – there are no distinctions. This is because
there are always some grey areas and hurdles to overcome and that means it can
be unpredictable. So therefore a lot of what governs and underpins your ability
to deliver successfully is the processes that prescribe how you identify,
resolve and future-proof these areas of uncertainty.
In the bad old days of waterfall
this uncertainty would always freak people out and because you released in a “big
bang”, the architecture and design tended to be gold-plated inside and out from
the start (whether it was always necessary to do this or not), and it would be
accompanied by endless reams of documentation that would get reviewed once, and
then get dropped somewhere in a CMS never to see the light of day again. The
really big waterfall projects would hardly ever make a delivery date due to the
big bang approach of deployments as getting it installed correctly in the
testing and production environments could waste days/weeks getting everything
configured and integrated. Then Agile came along and we started to see the
light that little bit better by releasing smaller and building integrity into
the product.
When application development work
started to get moved offshore in Australia around the early 2000’s, thanks to a
big push from the banks and the big telcos, every problem we faced doing
traditional waterfall got exacerbated because, despite the best efforts of the
onshore teams, the offshore teams would struggle to understand every
requirement due to the highly complex nature of the systems that had been
produced. Architecture gold-plating (because of the big bang unpredictability),
and the need to wade through all the documentation to understand what the hell
was going on, confused people a great deal. It would take months to understand it
all and the offshore teams were (stupidly) expected to come up to speed in a
matter of weeks – as promised by the glittering powerpoint presentations of
their high-end consultants. They never did of course – no-one could, and a big
part of the “sell” of these big offshoring companies, which was their sophisticated
training centres that claimed they could train up resources to meet demand anywhere
in the world, sounded as ridiculous now as it did then.
I worked on a project such as this once, here is my story
The architecture of the system I
worked on was highly complex. Ridiculously so. It was probably the worst
candidate for a system to be offshored that I have seen in all my years working
in IT. When the system was first transitioned offshore it collapsed in
spectacular fashion because the offshore teams did not have the domain nor the
extensive technology experience to implement or understand the system and how
to change, enhance and support it correctly. This caused all the delivery dates
for new initiatives to be missed, everyone worked late nights and after a few
months the hatred towards the company, management and the poor offshore
resources themselves (who were doing their best), by the onshore resources was
not even trying to be kept hidden. The client was furious at the missed dates and
the costs blew out badly. In this early stage numerous people had to travel to
India to rescue the project at great cost to the company. In short every
promise of improved quality, productivity and cost was broken.
As the onshore teams got so badly
burnt by the late nights making up for offshores mistakes, it decreased workplace
morale and to make matters worse, the company started cutting costs due to compensate
for the project cost blowouts as the schedules started to get missed as the
money wasn’t coming in. Even beers on Fridays were cancelled (in an Australian
work-force that is like inviting mutiny upon yourself), and morale got worse
and worse.
Down the rabbit hole of fail we go....
Because the offshore teams were
failing so badly, causing much finger pointing and blaming on both sides of the
fence, the architects and designers (under a considerable amount of pressure) started
to gold-plate their systems and designs even more to remove any traces of
ambiguity – and to cover themselves in the face of any management wrath. The
designs became so complex that one of the architects years later remarked, “we made
it so generic it actually became unusable”! As a result estimates to deliver
the software doubled, even tripled, to compensate for the delivery problems as
no-one was crazy enough to make promises based on the pre-offshoring days estimates.
The client started to get even
more upset at the cost blowouts (because they were promised things would get
cheaper) and managers, who had already worn out any goodwill to their
architects and tech-leads could not get anyone to budge on estimates so under
pressure they started over-promising to the clients hoping, praying, things
would improve.
And of course they didn’t.
Now in a total panic senior company
management implemented more knee-jerk responses to stem the cost bleeding because
they were losing control. To appease investors alarmed at the spiralling costs
they made even more cutbacks and produced lots of spin both internally and
externally to paint a better picture of the situation in an attempt to “sell”
the new offshore delivery model and pretend things were going well. This was of
course in total contrast to the experience on the ground and made the whole
situation even worse as it bred a massive amount of contempt amongst those
doing the work. Even anonymous blogs were created that mocked the company and
told many home truths that the company scrambled to get shut down and blocked
by ISP and network providers! The environment became incredibly toxic and
hostile and people left the company in droves fed up with how badly everything
got managed. Those that were left jumped back on the plane for extended stays
in India but then had to fight tooth and nail to claim back expenses incurred
as the company kept cutting more and more corners to compensate for
cost-blowouts requiring digging into more and more capital.
It was an absolute nightmare and
mis-managed from the top to the bottom. The initial cost-cutting set the scene
for failure and as we went further down the spiral more cost cutting just
exacerbated the issues.
So how do you get it to work?
There is a lot of literature and great publications now around the DevOps movement. A movement borne out of Agile to wholly integrate development and operations into a single functioning flow-through work process that enables fast and quick integration of code, automation of virtually everything and continuous one-click deployments to name but a few desirable capabilities. If you aren’t DevOps capable don’t even think about doing bespoke application delivery out of another country. For a start you can’t be as Agile as you can be onshore so you’re already at a disadvantage when it comes to team collaboration and working closely with the business. Also if offshore start spruiking the cheaper labour costs as a reason to not investing up-front in DevOps practices such as automation testing, code quality checking, continuously deployable pipelines and coded-infrastructure as they’ll just “assign cheaper resources to review and validate manually” then start scanning the job classifieds because you are going to go down.
Move it all to the cloud and
automate the shit out of it
You have to invest in your offshore delivery centres, moreso than onshore and you have to do it well. For a start avoid physical infrastructure and put all your development and testing in the cloud. Before you do that ensure all your servers and environments are coded up. That means you can restore and rebuild entire environments from the click of a button. It is not impossible, yes it will take time, but the long term benefits will protect and insure you against failure. This must be completed and in-place weeks before development even starts so the environments are completed, properly monitored, tested and cost-forecasted.
Hire properly offshore
No graduates.
If the people on your delivery project do not have a minimum of 5 years technical delivery experience then do not hire them. In some countries there is a culture of “it’s okay, they got good marks, they’ll learn as they go” and in IT this theory simply does not work. You need delivery experience, you need to be technically proficient and you need to have experience working with other cultures in other countries. Good people cost money, offshore is no different, but offshore it is even more critical because you really need smart and motivated people that can think on their feet.
Automate quality checking to the nth degree
And that means everything. Unit Test Code coverage above 90%, code metrics checking for cyclomatic complexity, lines of code, class coupling, maintainability and anything else you can throw at it. This all has to be automated on check-in. Furthermore database script changes should be tested against database rebuilds, selective automation test execution for critical end-to-end functions should also be thrown in.
Use code reporting tools in conjunction with build and code management software as much as you can. Produce reports and get them delivered to delivery managers daily so they can see how things are tracking.
Test the hell out of it often
Full suite of automation testing should run every night. As a minimum. Automation testers should start on day one as should DevOps people. No project should be without this. The capability to extend and support must work hand-in-glove with development and functional testing to amplify the feedback loops and keep everything rigorously tested
Get onshore and offshore
passports and visa’s ready
People need to travel and they need to travel often. If you want to have any hope of doing offshore delivery successfully you have to do this. There should be back and forth travel occurring every two weeks for the duration of the project. Extended stays should not happen, two-week stints are enough. This will ensure the teams are always working together and avoids the “us and them” situation from occurring. The more people work together, the better they work together and travel should be happening from the topmost level of PM’s through to the low level.
Do not avoid doing this. Do not think that travel at the start of the project to cement the understanding is all that is needed with travel at the end catered for to “bring it all home”. This does not work because there is all that time in the middle where the problems occur. I have seen this situation occur numerous times.
Summary
Cost-cutting up-front without investing properly in transitioning, training, infrastructure etc… is incredibly stupid when offshoring application development and it is even more disastrous when performed on projects in-flight when things start to go bad. Offshoring is not going to be cheap in the short term, and it is not going to be cheap in the long-term if you try and cut corners. You have to invest and invest heavily to get it to work from day zero in every area.
In my experience the developers,
testers, DevOps and architects (mostly) get the need for all this extra
overhead, both onshore and offshore. And it is not a one-off thing. Doing this is
a cycle of investment, from project to project, that must be repeated
constantly, to refine and enhance so that you delivery standards are
continuously improved. The first project will be the worst, it will cost you
way more than you planned but if you stick with it and maintain the investment
and support the costs will start to come down. You have to retain this focus,
the more you keep up the focus on ever improving quality the faster and faster
you will get. Doing bespoke application delivery between countries is not easy
and it is not, and never will be, fundamentally cheaper. As soon as you realise
this, and budget your projects accordingly to address it, you will have a hope
of succeeding.
