Why do developers and ops fight?

My question presupposes that they do. But I’ve worked at banks, Telcos, and software companies, and I’ve seen disagreements and conflict in each of these environments. Developers get frustrated at why it takes weeks to open a firewall port, which they know can be done in a few minutes or hours. Ops get frustrated at developers throwing new code over the fence, and then getting blamed if it doesn’t work. Developers resent ops saying “this is a coding/software problem” for things they perceive that ops should be able to sort out. Ops are sick of being asked to deploy code that was not designed to be easy to manage, and may not fit in very well with their runtime environment.

There are real costs to this kind of thing. Development teams don’t get what they need in a timely manner. Operations teams have people so stressed they are thinking of resigning. But the biggest problem of all – it’s a waste of time. Literally.

I think people who don’t have direct experience would be amazed at how much time developers in corporate environments waste getting infrastructure set up. If you’re lucky enough to have a lot of autonomy you might be able to set up your own automated test systems, but unless you run them on IT infrastructure you’re stuck with managing them yourselves. And that’s inefficient for the company as a whole. And IT operations can waste a lot of time trying to deploy apps that just don’t fit very well with their existing infrastructure and processes.

But it doesn’t need to be that way.

On the surface it looks like there’s a fundamental conflict in the motivation of devs vs ops. Developers are incentivized on getting changes and new features made to the software, and want to change things as quickly as possible. Ops are incentivized on stability of the environments and software they are managing.

But take a step back and in reality both groups actually care about the same thing – the applications that are being developed, deployed and managed. And the components (databases, web services, scripts, UI, source repositories) that make up those applications.

Of course each group has a different emphasis – devs care more about source control, agile task tracking, and who broke the build. Ops care more about monitoring, logs, and infrastructure. But both are dealing with the same applications, the same pieces of work. This is why ALM (Application Lifecycle Management) tools are experiencing growth at the moment. But they don’t go far enough.

So the obvious solution seems to be to create a single view of what is going on for developers and operations, based around what they both care about – the applications.

Bringing in information and events from a bunch of different systems can save people time by letting them see everything that relates to their daily work in the context of the applications they care about. These events can then form the context for discussions that can include both developers and operations. And once people from the two groups are talking it becomes a lot easier for them to work together both on a daily basis and during a crisis.

At the end of the day it’s all about the people. Developers and IT operations people just need a suitable context for collaboration to take place.

Appsecute started life as a management toolset for Platform as a Service (PaaS). As we’ve grown and adapted with our market and users, so too has our experience with PaaS.

On our journey we’ve experienced the pleasure and frustration that PaaS can deliver and have gained some interesting insights into how to use PaaS to deploy and manage applications. The promise of PaaS is real – it delivers significant cost and time savings, with a few caveats.

Our journey started with Cloud Foundry, a relatively young PaaS that has huge potential. Cloud Foundry is an open source platform built on Ruby. It was created by VMware which has since spun it off to a subsidiary as part of the Pivotal Initiative. Because Cloud Foundry is open source anyone is free to use it and provide commercial services around it – and because building a PaaS is hard (like, seriously hard) a number of companies have decided to leverage the existing work on Cloud Foundry and have used it to provide their own Cloud Foundry-compatible platforms. When we were looking at Cloud Foundry our software was built using .NET, which ironically Cloud Foundry doesn’t officially support. For .NET support we turned to Tier 3, a cloud computing company with a focus on the enterprise. Tier 3 added .NET support to Cloud Foundry through their open source Iron Foundry project, which they also use in-house to provide a more compelling enterprise proposition they call Web Fabric.

After using the Cloud Foundry platform provided by Tier 3 we learned our first lesson about PaaS – access to 24×7 support staff is an absolute necessity. By using PaaS we knowingly gave up the keys to the castle; we no longer concerned ourselves with the underlying infrastructure, operating systems or frameworks. This is a brilliant and highly efficient way of working right up to the point where you can’t do something that you absolutely need to do, and you’re not able to drop down to a lower level to achieve it.

One example of this was our requirement to deploy .NET based background tasks, a type of application not supported by Cloud Foundry even with Tier 3′s enhancements. Without the fantastic support provided by Tier 3 we would have found ourselves looking for a new home for our applications. Instead, Tier 3 worked with us (outside of normal business hours) to help us find a solution. In the end some clever networking rules meant we could deploy our .NET background tasks to virtual machines and have them talk to the MySQL database provided by the PaaS. This did mean we lost the utility of PaaS when it came time to deploy new versions of the background tasks but we achieved the outcome we wanted in a very small timeframe. This would not have been possible without the competent and genuinely helpful support engineers from Tier 3. Key takeaway: If you’re doing more than kicking the tires with PaaS look for a vendor that is willing to work with you to achieve business outcomes (Tier 3 and Apprenda stand out here).

The next stop on our journey is Nodejitsu – a somewhat interesting twist considering we had previously been using .NET. If you have been following Appsecute you will have seen that a few months ago we set out in a new direction to build collaborative application management tools for developers and operations. As part of this shift in focus we had the opportunity to pick a new language to build the new tools in. We had been growing increasingly frustrated with .NET and the hoops we had to jump through to use it to build the modern API-driven web services that we were building. After reviewing our options and working on a few prototypes we settled on Node.js (JavaScript) because we were absolutely blown away by how productive it was. We were literally adding new features in days that would have taken weeks in .NET.

We set out to find a platform to host our Node.js code. Nodejitsu stood out to us as our new direction was very real-time and had a hard dependency on WebSocket support, which most platforms have been slow to adopt. Our initial experience with Nodejitsu was great; their tools are very streamlined and slotted into our Node.js development environment without any hassle. We quickly deployed our new applications to Nodejitsu (leaving the legacy .NET apps on Tier 3) and found their WebSocket support to be great.

Unfortunately, we soon started to run in to some problems with Nodejitsu. Deployments of new application versions would sometimes fail multiple times in a row, and each failure would be due to a different error, giving the impression that there were some much bigger problems under the hood. We persevered and worked with Nodejitsu to resolve this – we were aware that some of these issues were due to roll-out of new data center locations by Nodejitsu. We could have put the deployment issues down to teething problems and hung in there while they were resolved, but we have certain applications that often need to be deployed in parallel as one depends on the other (e.g. our client web app depends on the API provided by our server, if the API changes the client needs to be updated) so sometimes we would find ourselves in the situation where one app would deploy successfully on the first go, and then ten minutes later we would still be attempting to deploy the second app. This left our live site vulnerable to issues where client and server didn’t match during this gap in time. And then we started running in to seemingly random errors at runtime where our apps would just stop responding, and then magically start working again some time later.

We love what Nodejitsu is doing – they are a young company building innovative cutting-edge infrastructure. But our experiences made it clear that we needed the enterprise-grade support provided by Tier 3. This isn’t a criticism of Nodejitsu – I must stress they were very helpful and willing to engage with us. It’s more a reflection of the maturity of their platform at the time we happened to start using them. They have many customer success stories, we just weren’t one of them. Key takeaway:  The same as before, find a vendor that understands your needs and desired business outcomes and will work with you. So we felt we had to move on. Next stop, Heroku.

Before I talk about our experience with Heroku (it’s been great) I want to point something out – look how fluid we are with the PaaS vendor we use. It’s quite subtle but working with PaaS means working with higher levels of abstractions, and PaaS is mature enough that the abstraction is fairly consistent across all platforms. It is trivial to move between PaaS vendors, in the public cloud at least. Moving 8 applications from Nodejitsu to Heroku took a couple of hours one afternoon. The apps were deployed to Heroku and tested before a simple DNS tweak cut over traffic from Nodejitsu to Heroku, with zero downtime. We didn’t even need to touch our databases as they’re hosted with MongoLab in the same city as both Nodejitsu and Heroku (and AWS, Rackspace, and others). One of the promises of PaaS is no vendor lock-in, and it’s a reality, especially if you decouple your database from your PaaS hosting.

Since moving to Heroku we have had a rock stable experience. Heroku’s deployment mechanism is based on git, which is extremely reliable even on a slow internet connection (we frequently deploy changes on the road). Heroku’s mature add-on marketplace has also made setting up tools like New Relic an absolute breeze. We have had to make one compromise with Heroku that I’m still uneasy about – most platforms have been slow to adopt support for WebSockets and Heroku is no exception. To use socket.io on Heroku we had to configure it to use long-polling instead of real WebSockets, which brings with it significant performance penalties and reliability problems. We hope Heroku will announce WebSocket support soon. The key takeaway from our time with Heroku has been that using a mature platform has so far negated the need for access to 24×7 support. The web is filled with so many examples, workarounds and documentation for Heroku that we have been able to resolve any problems with a quick Google search.

That’s been our journey so far and I’m still amazed at how trivial it is to move applications between PaaS vendors. There really is no lock-in with PaaS which puts us (the customer) in a great position.

In my next blog post I want to talk about how we decoupled our databases from our PaaS hosting using MongoLab, as it is the missing piece of the no lock-in puzzle.

Thanks to our good friends at NZTE, Appsecute has been given the opportunity to pitch at the America’s Cup Investment Showcase in San Francisco in September.

This is perfect timing for us as it is when we had planned to start entering discussions for a series A round to help Appsecute grow to become the platform of choice for developers and IT operations. We’re no strangers to these pitching excursions, having been finalists at VentureBeat’s Cloud Beat Innovation Showdown a few months ago. We’re getting good at this!

Don’t you just love the warm fuzzies from this email…

Congratulations! You have been accepted to participate in the America’s Cup Investment Showcase. Your company was selected as being one of the best and most innovative new companies coming out of New Zealand and will have the opportunity to present to some of the most influential investors in Silicon Valley.

We have deployed a new connector that supports Heroku Deploy Hooks, so you can now be told when team members deploy new versions of your applications!

To use the new connector head over to the connections page, clicked ‘Connect a Service’ and select the Heroku Deploy Hooks connector, then follow the simple instructions here to configure the Deploy Hooks add-on in Heroku and this is the result:

Heroku Deploy Hooks Connector: Be notified when team members deploy application changes

The code for the Heroku Deploy Hooks connector is open source on GitHub here: Heroku Deploy Hooks Connector

Recently we announced a new vision for Appsecute –  you can read about it here. As this is such a big change for us I wanted to talk about how and why we got to this point and what it means for the future.

When we started working on Appsecute the PaaS market was very immature, which is arguably still true today, and having come from an enterprise operations background we saw an opportunity to build out a layer of enterprise class features on top of PaaS – we could bring standardized identity management, fine grained access control, audit, deployment workflow and recovery etc. to PaaS. And not just to one platform, but to them all. The end goal was a product that would no doubt be useful to individuals and teams, but where it would really shine would be in large scale deployments where hundreds or even thousands of applications were being managed.

This end goal influenced the way we built Appsecute. We first thought of the enterprise use case for a feature before we looked at how it would be used by developers or operations folk on a smaller scale – as individuals or within a team managing one application. In spite of that we actually found the DevOps people were the ones who started using it – PaaS in the enterprise isn’t there yet.

It’s likely, in my opinion, that in 3 to 5 years time enterprises will have large scale deployments of PaaS, where green-field applications are being built exclusively on PaaS and legacy applications are slowly being migrated where possible – the value that PaaS delivers to the enterprise is massive. The problem with this vision is that it’s ‘in the future’, it’s not here and now.

As a startup we need to find a viable and sustaining market today – paraphrasing John Dillion (CEO of Engine Yard) at CloudBeat last year, the existing PaaS providers are currently eking out an existence in a new market, everyone is trying to find their niche to sustain themselves, helping to grow the pie for everyone and trying to ensure they’re around to reap the rewards when PaaS goes mainstream.

The problem with our approach is that we had not identified a viable niche to sustain ourselves here and now. Don’t get me wrong, Appsecute has hundreds of active users and is growing every day, but from talking to these users it’s clear they are not enterprise customers managing hundreds of applications, they’re working on a smaller scale managing a few apps. At this smaller scale Appsecute had a less compelling value proposition.

So we found ourselves serving two markets, the first being the enterprise users that the features were built for and the second being the DevOps users that were finding value in our toolset. The market had spoken! After evaluating our options and sales strategy we have decided we wanted to make sure that there were compelling reasons for DevOps to want to use our product every day. We believe that Enterprise adoption will come, but will start with bottom-up use by the DevOps users.

It was a hard decision to make. We loved the thought of adding features focused specifically on DevOps but we also wanted to change the whole emphasis of the product to work more smoothly for DevOps and small teams. To do this properly we had to go back to the drawing board on some fundamental pieces of Appsecute – we needed a new user model, a new access control model, a more flexible organization model etc. We found ourselves reworking many of the internals that we’d worked so hard to model after the enterprise. But after the remodeling we found something interesting – we had a system that worked not only for small teams, but was actually better than the old model for enabling collaboration in the enterprise.

It also meant finding a real and compelling value proposition to make sure that DevOps would want to use the product every day. To do this we talked to existing users, looked at all the metrics we had recorded, looked at the marketing we had done and scrutinized the messaging that had driven user adoption among DevOps. We also turned to ourselves as we’re very much our own target market now, so we asked what would be useful for us.

If you read the outline of the new vision linked to above, you’ll see the new direction is focused on managing a system or a project in a collaborative and centralized way, one where people and software take part in a social stream of the day to day activities involved in DevOps. PaaS is still a very important piece of the puzzle but it’s no longer the only piece. It’s now on an equal footing with source control, support systems, bug tracking, monitoring, and social interactions, among other things.

It was totally necessary to reevaluate our product strategy and we strongly believe in the new direction but as a founder, as someone that’s dedicated almost a year building out our code base, it was painful and humbling to face up to the fact that we hadn’t got it ‘right’ the first time around. It was especially hard because I still believe so strongly in the original vision and value proposition, but it’s just too early, the enterprise market isn’t ready yet.

With all that said we’re extremely excited about the new direction. It has amazing potential to streamline the day to day work of DevOps and stays true to our original theme of a single pane of glass to manage cloud applications.

Today we opened the doors to the public beta of the new Appsecute – a broader and more collaborative extension of the “old” Appsecute, we’ve gone beyond focusing exclusively on Platform as a Service and extended our single pane of glass to the cloud in general.

The new Appsecute pairs a social stream with a dashboard of application health to create the ultimate real-time application management tool for developers and operations. The social stream shows events from all the services you use (such as GitHub, Zendesk and PagerDuty etc.) and allows you to comment and coordinate work directly on the stream – think of it like a specialized Facebook-style timeline.

We’re excited by the possibilites this way of working enables, the feedback from private beta users has been that Appsecute quickly became the one tool everyone had up on their screen all day to stay on top of things in real-time.

Expect some exciting announcements over the coming weeks as we rapidly iterate and improve the existing features while adding support for new services (AWS, New Relic and TravisCI are in our sites).

Check out the new Appsecute: https://appsecute.com

Today we’re really excited to outline a new vision for Appsecute – it’s social, it’s open, it’s collaborative and it’s extremely addictive!

Ever had that problem where you’re using a dozen different services to manage your application? And you have to log in to them and watch them every day just to stay on top of things? You know what I mean… GitHub for source control, Zendesk for support, Heroku for hosting, TeamCity for continuous integration, Loggly for logs, email threads for discussions, something else for instant messaging… the list goes on.

Let me introduce the solution to this problem – the new Appsecute, the social stream for developers and operations:

Appsecute started out giving you a single place to manage all your PaaS applications, it aggregated info from all the different platforms you might be using and gave you a real time view of application health. It had the concept of organizations and teams, and had a fine grained access control layer to protect production applications.

Now, we’re going beyond PaaS, we’re bringing the single pane of glass and single place of management to the cloud in general – Appsecute is becoming the place you go to every day to stay on top of everything.

Launching in a few weeks, the new Appsecute is a social stream that understands your systems and the software components that make up those systems.

It knows that your system includes some repositories on GitHub, it knows you’re using CircleCi for continuous integration, it knows the production version of your app is running on Cloud Foundry or Heroku and it knows you’re using Zendesk for support. In fact, through our API, it knows about everything that you as DevOps care about.

The real-time stream shows events that are occurring in the services you use and encourages collaboration around those events. So what is an event? An event can be anything – someone pushing code to GitHub, a build failing, tests failing, an application going offline, a support request coming in, a daily summary of the metrics you care about – literally anything. And you and your team can write comments directly against any event, kind of like Facebook for DevOps. It’s a great way to agree what to do about issues that come up, and who is going to take action.

Events are smart, too. We can use them to alert you to issues and tell you once they’ve been resolved. In the screenshot above, to the right of the timeline, you can see all the components you’ve associated with your system – and we can let you know if they’re healthy or not. If a build fails or monitoring detects an issue with a production application, that component will go red until the problem is resolved.

We have an open API that allows anyone to plug a new system in to the social stream – ‘connectors’ can be written and published on Appsecute so that others can use them. And if anyone in the community writes a connector and open sources it we will host it for you!

We’re launching with some connectors we’ve written ourselves – GitHub, Zendesk, CircleCi, Cloud Foundry, AppFog and Heroku – and we will be working hard to keep a constant stream of new connectors coming out the door after launch.

Right now we’re running a private beta of the new Appsecute. If you’re interested in getting in the beta please comment or email me at [email protected]