Credits: Computerweekly
Software development practices in the enterprise have traditionally focused on delivery high-quality code built on proven platforms. But the web and the emergence of apps, built on web scale infrastructure, are rewriting the rulebook.
In fact, businesses struggle to compete with startups that can somehow maximise the value of the new economy, and are able to undermine traditional business models.
“Over the past 20 years, IT has been set up for efficiency, cost reduction and doing things as safely as possible,” says Benjamin Wootton, co-founder of Contino.
He says companies are now driven by the need to work faster and are becoming more agile in order to improve the customer experience.
“This is applying pressure on IT and how we develop software,” he says.
Whereas IT heads previously implemented heavyweight internal IT processes and used outsourcing to reduce cost and maintain quality, in Wootton’s experience, this style of running IT slowed down IT departments. “DevOps and continuous delivery allow organisations to operate faster, which is what enterprises want to do today,” he says.
But Wootton argues that among the challenges for IT leaders is the fact that big enterprises are risk adverse and tend to stick with a tried and trusted formula, often contrary to contemporary best practices.
Shifting the enterprise mindset
Kingsley Davis, a partner at Underscore Consulting, adds: “You want to deliver quickly and at pace, which means having a clear strategy about what things are not important for the product.”
Technology such as Docker, to enable developers to create code that can run in their own containers, along with the ability to have short feedback loops, helps businesses to adapt more quickly. Such technology and techniques form the basis of the cultural shift that companies of all sizes need to make to enable their developer teams to become more adept at delivering software quickly, says Davis.
“Culture is very easy to instil when there is a small group of people,” he says. “Hiring is key.”
Davis recommends that IT leaders plan in advance, and hire people appropriate to the direction the IT strategy is taking.
Russ Miles, lead engineer at Atomist, believes IT leaders can learn much from the way webscale organisations approach software development. “Organisations of any size have to compete,” he says.
The speed of change is such that IT leaders cannot afford not to adapt their business processes. “People look at what Netflix is doing and the thing to take away is that agile software development will only get you so far,” says Miles. “The software itself needs to be as adaptable as the process.”
What this boils down to, says Miles, is that IT leaders need to figure out how to adapt systems and the work IT departments need to do, to achieve the speed and flexibility required by the business.
A case for smarter analytics
If they cannot meet the needs of the business, business users will go elsewhere, or even develop the systems themselves.
“Business users are driving software development,” says Frank Ketelaars, big data technical leader for Europe at IBM. This is a form of shadow IT, he adds. “They use spreadsheet data warehouses as their own analytics platform.”
Also, the availability of deep learning services is pushing the boundaries of analytics in terms of the massive computational intelligence such algorithms can bring to bear on hard-to-solve problems. But with such technological developments comes new challenges.
What developers need, says Ketelaars, is a route to analytics services via a common programming interface, giving them the features of an analytics tool within their own applications.
Another challenge facing analytics applications is how to make sense of the data. “Deep learning is here, but one thing that is missing is context,” says Ketelaars. “If I have a picture with two people running after each other, and one has a frisbee, I know this image is about playing.”
Deep learning algorithms can instantly recognise the image of two people running, he says, but adds: “The context changes dramatically if one of them has a chainsaw. This is where context controls what you should do with the image.”
For Ketelaars, understanding context will be a key requirement in applications, to understand the meaning behind the analytics. “You have to start thinking about what data you have to control the behaviour of your application,” he says.
Essentially, applications become smarter, providing users with the information they need based on a deep contextual understanding of what it deems relevant or important.
Improving tooling
Arguably, there is room for improvement among the array of tools, building blocks and techniques that developers use to create software, says Phil Trelford, founding member of #F Foundation. “A general-purpose programming language is a bit like a spanner and we are all trying to build large systems with spanners,” he says. “What I would like to see is precision tools.”
In fact Trelford goes further, saying the industry needs better “meta tools”, in other words software tools to help the developers of programming tools build precision instruments, rather than generic spanners.
While, as the saying goes, “a bad workman blame his tools”, software developers are keen to see improvements in the tooling they use. In part, this helps them cope with the added complexities when coding and operations become one, as in DevOps, says Trelford.
“Personally, I would like to be able to say, ‘I need to make this thing happen’, then make it as quickly as possible,” says Miles. But today, developers need to draw together a lot of threads to create and run applications successfully, and complexity is increasing all the time, he says. “Smart tooling will help them handle the cognitive overhead,” he adds.
Enterprises appear to be following smaller companies in adopting new, more productive ways to code. For instance, WhatsApp, which was developed by a handful of Erlang developers, was sold to Facebook for $19bn, while Walmart recently acquired the F#-based Jet.com e-commerce platform for $3bn.
While procedural programming languages such as Java and C have been the bread and butter of enterprise software development, what has been particularly interesting for software development is the uptake of functional programming in recent years, particularly among companies that need to support large numbers of internet customers.
As Trelford explains: “Apart from Java, which has huge user groups, the biggest programmer user groups in London are those of the functional programming languages. I run the functional London meeting here. We have been meeting here for about six years and we have over 1,000 members. Scala, I believe, has 1,500-2,000 members and Clojure has been growing, as has Erlang.”
Many proponents of these programming languages talk about how little code they need compared with using a procedural language, which makes them attractive for writing code quickly, says Davis.
Sparks argues that, ideally, software developers should look at architecting systems by integrating many small software components, each of which has been developed to be highly reliable. “We are moving to a world where individual software systems are becoming so critical and so important that we had better build them using the best tools and the best languages, and this is why we are seeing a rise in interest in functional programming,” he says. “Functional programming is a better approach for writing highly reliable systems.”
Changing software development landscape
Among the changes Miles is seeing is that software development is no longer a factory floor to churn out new products. Rather, he believes software development is evolving into a continuous R&D practice.
“Companies that regard software as a driver for them are the ones that will win and one of the pieces of advice I give to company boards is that they should not think of software development as a general problem that we can solve by throwing more people at it,” says Miles. “Think of software development as a place where you might be surprised what comes out.”
Enterprises do not often have the luxury of greenfield development, but as Trelford points out, where enterprises need a new system, there is the opportunity to experiment with the least risk.
Wootton says: “Everyone is always excited about the new greenfield stuff, but there is a real business case with legacy.” Often, the real business case is actually the J2EE or .Net code that has been running for a decade or more and requires a big support team.
“You might do something crazy like services on your mainframe, but it turns out this may be where you get the biggest return on investment,” he says.
“Legacy is a bad thing,” says Sparks. While it is exciting to create new code based on microservices and perhaps functional programming, the biggest challenge faced by corporate IT is often how to handle a growing legacy of old stuff.
“It can be the millstone that drags you to the bottom of the sea,” says Sparks, who urges CIOs to look constantly at what can be decommissioned, and have development teams write new applications. These not only help to move the business forward but, at the same time, enable IT to decommission something else.
Davis believes that the new techniques available to developers, such as reusable microservices, containers, functional programming and continuous delivery, offer enterprises an ideal opportunity to reduce risk and improve reliability.
“It is all about safe, small-scale scalability,” he says. The tools and techniques discussed enable IT departments to avoid the risk of modifying mission-critical applications by augmenting them in a highly controlled way, adds Davis.