We have just deployed an update that adds support for basic user accounts to Appsecute. You can now register an account with Appsecute using a username and password, as well as using our existing OAuth providers if you would prefer.
This is a step towards enabling private deployments of Appsecute behind the firewall, for use with your internal systems.
My question presupposes that they do. But I’ve worked at banks, Telcos, and software companies, and I’ve seen disagreements and conflict in each of these environments. Developers get frustrated at why it takes weeks to open a firewall port, which they know can be done in a few minutes or hours. Ops get frustrated at developers throwing new code over the fence, and then getting blamed if it doesn’t work. Developers resent ops saying “this is a coding/software problem” for things they perceive that ops should be able to sort out. Ops are sick of being asked to deploy code that was not designed to be easy to manage, and may not fit in very well with their runtime environment.
There are real costs to this kind of thing. Development teams don’t get what they need in a timely manner. Operations teams have people so stressed they are thinking of resigning. But the biggest problem of all – it’s a waste of time. Literally.
I think people who don’t have direct experience would be amazed at how much time developers in corporate environments waste getting infrastructure set up. If you’re lucky enough to have a lot of autonomy you might be able to set up your own automated test systems, but unless you run them on IT infrastructure you’re stuck with managing them yourselves. And that’s inefficient for the company as a whole. And IT operations can waste a lot of time trying to deploy apps that just don’t fit very well with their existing infrastructure and processes.
But it doesn’t need to be that way.
On the surface it looks like there’s a fundamental conflict in the motivation of devs vs ops. Developers are incentivized on getting changes and new features made to the software, and want to change things as quickly as possible. Ops are incentivized on stability of the environments and software they are managing.
But take a step back and in reality both groups actually care about the same thing – the applications that are being developed, deployed and managed. And the components (databases, web services, scripts, UI, source repositories) that make up those applications.
Of course each group has a different emphasis – devs care more about source control, agile task tracking, and who broke the build. Ops care more about monitoring, logs, and infrastructure. But both are dealing with the same applications, the same pieces of work. This is why ALM (Application Lifecycle Management) tools are experiencing growth at the moment. But they don’t go far enough.
So the obvious solution seems to be to create a single view of what is going on for developers and operations, based around what they both care about – the applications.
Bringing in information and events from a bunch of different systems can save people time by letting them see everything that relates to their daily work in the context of the applications they care about. These events can then form the context for discussions that can include both developers and operations. And once people from the two groups are talking it becomes a lot easier for them to work together both on a daily basis and during a crisis.
At the end of the day it’s all about the people. Developers and IT operations people just need a suitable context for collaboration to take place.
Appsecute started life as a management toolset for Platform as a Service (PaaS). As we’ve grown and adapted with our market and users, so too has our experience with PaaS.
On our journey we’ve experienced the pleasure and frustration that PaaS can deliver and have gained some interesting insights into how to use PaaS to deploy and manage applications. The promise of PaaS is real – it delivers significant cost and time savings, with a few caveats.
Our journey started with Cloud Foundry, a relatively young PaaS that has huge potential. Cloud Foundry is an open source platform built on Ruby. It was created by VMware which has since spun it off to a subsidiary as part of the Pivotal Initiative. Because Cloud Foundry is open source anyone is free to use it and provide commercial services around it – and because building a PaaS is hard (like, seriously hard) a number of companies have decided to leverage the existing work on Cloud Foundry and have used it to provide their own Cloud Foundry-compatible platforms. When we were looking at Cloud Foundry our software was built using .NET, which ironically Cloud Foundry doesn’t officially support. For .NET support we turned to Tier 3, a cloud computing company with a focus on the enterprise. Tier 3 added .NET support to Cloud Foundry through their open source Iron Foundry project, which they also use in-house to provide a more compelling enterprise proposition they call Web Fabric.
After using the Cloud Foundry platform provided by Tier 3 we learned our first lesson about PaaS – access to 24×7 support staff is an absolute necessity. By using PaaS we knowingly gave up the keys to the castle; we no longer concerned ourselves with the underlying infrastructure, operating systems or frameworks. This is a brilliant and highly efficient way of working right up to the point where you can’t do something that you absolutely need to do, and you’re not able to drop down to a lower level to achieve it.
One example of this was our requirement to deploy .NET based background tasks, a type of application not supported by Cloud Foundry even with Tier 3′s enhancements. Without the fantastic support provided by Tier 3 we would have found ourselves looking for a new home for our applications. Instead, Tier 3 worked with us (outside of normal business hours) to help us find a solution. In the end some clever networking rules meant we could deploy our .NET background tasks to virtual machines and have them talk to the MySQL database provided by the PaaS. This did mean we lost the utility of PaaS when it came time to deploy new versions of the background tasks but we achieved the outcome we wanted in a very small timeframe. This would not have been possible without the competent and genuinely helpful support engineers from Tier 3. Key takeaway: If you’re doing more than kicking the tires with PaaS look for a vendor that is willing to work with you to achieve business outcomes (Tier 3 and Apprenda stand out here).
We set out to find a platform to host our Node.js code. Nodejitsu stood out to us as our new direction was very real-time and had a hard dependency on WebSocket support, which most platforms have been slow to adopt. Our initial experience with Nodejitsu was great; their tools are very streamlined and slotted into our Node.js development environment without any hassle. We quickly deployed our new applications to Nodejitsu (leaving the legacy .NET apps on Tier 3) and found their WebSocket support to be great.
Unfortunately, we soon started to run in to some problems with Nodejitsu. Deployments of new application versions would sometimes fail multiple times in a row, and each failure would be due to a different error, giving the impression that there were some much bigger problems under the hood. We persevered and worked with Nodejitsu to resolve this – we were aware that some of these issues were due to roll-out of new data center locations by Nodejitsu. We could have put the deployment issues down to teething problems and hung in there while they were resolved, but we have certain applications that often need to be deployed in parallel as one depends on the other (e.g. our client web app depends on the API provided by our server, if the API changes the client needs to be updated) so sometimes we would find ourselves in the situation where one app would deploy successfully on the first go, and then ten minutes later we would still be attempting to deploy the second app. This left our live site vulnerable to issues where client and server didn’t match during this gap in time. And then we started running in to seemingly random errors at runtime where our apps would just stop responding, and then magically start working again some time later.
We love what Nodejitsu is doing – they are a young company building innovative cutting-edge infrastructure. But our experiences made it clear that we needed the enterprise-grade support provided by Tier 3. This isn’t a criticism of Nodejitsu – I must stress they were very helpful and willing to engage with us. It’s more a reflection of the maturity of their platform at the time we happened to start using them. They have many customer success stories, we just weren’t one of them. Key takeaway: The same as before, find a vendor that understands your needs and desired business outcomes and will work with you. So we felt we had to move on. Next stop, Heroku.
Before I talk about our experience with Heroku (it’s been great) I want to point something out – look how fluid we are with the PaaS vendor we use. It’s quite subtle but working with PaaS means working with higher levels of abstractions, and PaaS is mature enough that the abstraction is fairly consistent across all platforms. It is trivial to move between PaaS vendors, in the public cloud at least. Moving 8 applications from Nodejitsu to Heroku took a couple of hours one afternoon. The apps were deployed to Heroku and tested before a simple DNS tweak cut over traffic from Nodejitsu to Heroku, with zero downtime. We didn’t even need to touch our databases as they’re hosted with MongoLab in the same city as both Nodejitsu and Heroku (and AWS, Rackspace, and others). One of the promises of PaaS is no vendor lock-in, and it’s a reality, especially if you decouple your database from your PaaS hosting.
Since moving to Heroku we have had a rock stable experience. Heroku’s deployment mechanism is based on git, which is extremely reliable even on a slow internet connection (we frequently deploy changes on the road). Heroku’s mature add-on marketplace has also made setting up tools like New Relic an absolute breeze. We have had to make one compromise with Heroku that I’m still uneasy about – most platforms have been slow to adopt support for WebSockets and Heroku is no exception. To use socket.io on Heroku we had to configure it to use long-polling instead of real WebSockets, which brings with it significant performance penalties and reliability problems. We hope Heroku will announce WebSocket support soon. The key takeaway from our time with Heroku has been that using a mature platform has so far negated the need for access to 24×7 support. The web is filled with so many examples, workarounds and documentation for Heroku that we have been able to resolve any problems with a quick Google search.
That’s been our journey so far and I’m still amazed at how trivial it is to move applications between PaaS vendors. There really is no lock-in with PaaS which puts us (the customer) in a great position.
In my next blog post I want to talk about how we decoupled our databases from our PaaS hosting using MongoLab, as it is the missing piece of the no lock-in puzzle.
Thanks to our good friends at NZTE, Appsecute has been given the opportunity to pitch at the America’s Cup Investment Showcase in San Francisco in September.
This is perfect timing for us as it is when we had planned to start entering discussions for a series A round to help Appsecute grow to become the platform of choice for developers and IT operations. We’re no strangers to these pitching excursions, having been finalists at VentureBeat’s Cloud Beat Innovation Showdown a few months ago. We’re getting good at this!
Don’t you just love the warm fuzzies from this email…
Congratulations! You have been accepted to participate in the America’s Cup Investment Showcase. Your company was selected as being one of the best and most innovative new companies coming out of New Zealand and will have the opportunity to present to some of the most influential investors in Silicon Valley.
We have deployed a new connector that supports Heroku Deploy Hooks, so you can now be told when team members deploy new versions of your applications!
To use the new connector head over to the connections page, clicked ‘Connect a Service’ and select the Heroku Deploy Hooks connector, then follow the simple instructions here to configure the Deploy Hooks add-on in Heroku and this is the result:
Heroku Deploy Hooks Connector: Be notified when team members deploy application changes
When we launched the new Appsecute two weeks ago we were flooded with requests to build connectors for peoples favorite systems. One of the most common requests that just kept coming up was for Travis CI, an open source continuous integration service that is integrated with GitHub (it also builds and runs tests for open source projects for free!). Initially we had planned to get to work straight away on an Amazon AWS connector, but since Travis CI kept coming up we had a look at what it would take to do the integration – we were happy to find that thanks to Travis CI’s open and well documented API it would be a relatively easy piece of work.
I personally launched straight in to the project and with a little help from the fantastic people in the Travis CI irc room we had the Travis CI connector up and running within hours, we even documented the whole process here as a tutorial to show other developers how simple it is to build connectors for Appsecute.
We’re happy to announce the Travis CI connector is now publicly available for everyone to use on Appsecute, simply head over to the connectors page, click the ‘Connect a Service’ button and select the Travis CI connector, within seconds the results of your Travis CI builds will be showing up on the Appsecute timeline. A failed build will cause the application status to go red to alert you to the issue and once the build is fixed everything will go green again, to let you know you can rest easy (or not, as the whole team will be able to see who broke the build!).
It was a great experience working with the Travis CI team, they’ve made a fantastic job of their API, making integrations like this possible. If you are a developer and haven’t heard of Travis CI before you owe it to yourself to check out their great service - https://travis-ci.org
The Travis CI team also provides a hosted ‘Pro’ version which is capable of building and testing your private GitHub repositories. You can sign up for the beta of the pro version here - http://beta.travis-ci.com. If you would like an early invite to the pro version then simply email [email protected] and mention Appsecute, the Travis CI team will help you out!
Recently we announced a new vision for Appsecute – you can read about it here. As this is such a big change for us I wanted to talk about how and why we got to this point and what it means for the future.
When we started working on Appsecute the PaaS market was very immature, which is arguably still true today, and having come from an enterprise operations background we saw an opportunity to build out a layer of enterprise class features on top of PaaS – we could bring standardized identity management, fine grained access control, audit, deployment workflow and recovery etc. to PaaS. And not just to one platform, but to them all. The end goal was a product that would no doubt be useful to individuals and teams, but where it would really shine would be in large scale deployments where hundreds or even thousands of applications were being managed.
This end goal influenced the way we built Appsecute. We first thought of the enterprise use case for a feature before we looked at how it would be used by developers or operations folk on a smaller scale – as individuals or within a team managing one application. In spite of that we actually found the DevOps people were the ones who started using it – PaaS in the enterprise isn’t there yet.
It’s likely, in my opinion, that in 3 to 5 years time enterprises will have large scale deployments of PaaS, where green-field applications are being built exclusively on PaaS and legacy applications are slowly being migrated where possible – the value that PaaS delivers to the enterprise is massive. The problem with this vision is that it’s ‘in the future’, it’s not here and now.
As a startup we need to find a viable and sustaining market today – paraphrasing John Dillion (CEO of Engine Yard) at CloudBeat last year, the existing PaaS providers are currently eking out an existence in a new market, everyone is trying to find their niche to sustain themselves, helping to grow the pie for everyone and trying to ensure they’re around to reap the rewards when PaaS goes mainstream.
The problem with our approach is that we had not identified a viable niche to sustain ourselves here and now. Don’t get me wrong, Appsecute has hundreds of active users and is growing every day, but from talking to these users it’s clear they are not enterprise customers managing hundreds of applications, they’re working on a smaller scale managing a few apps. At this smaller scale Appsecute had a less compelling value proposition.
So we found ourselves serving two markets, the first being the enterprise users that the features were built for and the second being the DevOps users that were finding value in our toolset. The market had spoken! After evaluating our options and sales strategy we have decided we wanted to make sure that there were compelling reasons for DevOps to want to use our product every day. We believe that Enterprise adoption will come, but will start with bottom-up use by the DevOps users.
It was a hard decision to make. We loved the thought of adding features focused specifically on DevOps but we also wanted to change the whole emphasis of the product to work more smoothly for DevOps and small teams. To do this properly we had to go back to the drawing board on some fundamental pieces of Appsecute – we needed a new user model, a new access control model, a more flexible organization model etc. We found ourselves reworking many of the internals that we’d worked so hard to model after the enterprise. But after the remodeling we found something interesting – we had a system that worked not only for small teams, but was actually better than the old model for enabling collaboration in the enterprise.
It also meant finding a real and compelling value proposition to make sure that DevOps would want to use the product every day. To do this we talked to existing users, looked at all the metrics we had recorded, looked at the marketing we had done and scrutinized the messaging that had driven user adoption among DevOps. We also turned to ourselves as we’re very much our own target market now, so we asked what would be useful for us.
If you read the outline of the new vision linked to above, you’ll see the new direction is focused on managing a system or a project in a collaborative and centralized way, one where people and software take part in a social stream of the day to day activities involved in DevOps. PaaS is still a very important piece of the puzzle but it’s no longer the only piece. It’s now on an equal footing with source control, support systems, bug tracking, monitoring, and social interactions, among other things.
It was totally necessary to reevaluate our product strategy and we strongly believe in the new direction but as a founder, as someone that’s dedicated almost a year building out our code base, it was painful and humbling to face up to the fact that we hadn’t got it ‘right’ the first time around. It was especially hard because I still believe so strongly in the original vision and value proposition, but it’s just too early, the enterprise market isn’t ready yet.
With all that said we’re extremely excited about the new direction. It has amazing potential to streamline the day to day work of DevOps and stays true to our original theme of a single pane of glass to manage cloud applications.
So you’re using Appsecute, receiving events in real-time, collaborating on the timeline, monitoring your applications and you’re finding it to be a great way of working.
But something is missing.
When building a connector we first need to establish a few things:
1. What OAuth2 server will be used for authentication? If the service you’re integrating with supports OAuth2 then use that, otherwise we recommend you use Appsecutes’ OAuth2 server. For this example we will use GitHub’s OAuth2 server as according to the TravisCi docs that is what they delegate to.
2. Does the service you’re integrating with have an API? If so, does it have a logical unit that can be mapped to Appsecute components? In Appsecute a component is a piece of a bigger system that makes up an application, connectors can expose the concepts within the service as components within Appsecute. For example the GitHub connector exposes repositories as components, and our PaaS connectors expose hosted application as components. In TravisCi, source repositories seem like the logical choice to map to components, and lucky for us TravisCi has an API that can list repositories.
3. How will you get events out of the service you’re integrating with? Webhooks are a great way to do this if the service supports them, otherwise you can fall back to whatever type of notifications the service uses (like email or Jabber) and finally if worse comes to worst you can poll the service periodically, although we would encourage you to ask the service provider to get their act together and provide notifications. TravisCi again makes this easy as they support Webhooks.
To get us started I’ve created a basic Node.js application using express for the web framework, this takes care of our routing and web services. I won’t go in to the details as it’s just a couple of files to start the server up on port 80 and to connect to the database (MongoDB for this example).
The interesting stuff comes next… Lets cover authentication first, when building Appsecute we made the decision that we would take care of as much of the authentication process as possible. The solution we came up with is that Appsecute will authenticate to the connectors OAuth2 server on your behalf, passing the resulting OAuth2 access token to the connector during API calls. What this means is that you don’t have to write the code to authenticate to GitHub’s OAuth2 server as Appsecute will do it for you. What we do need to do however is provide Appsecute the details it needs to authenticate so lets head over to GitHub and register a new OAuth application.
When registering OAuth applications (clients) for your connectors with the OAuth server the callback url should be set to “https://api.appsecute.com/connectors”:
Now that we have registered the application we have the client id and client secret that we will need to provide to Appsecute later. So that’s authentication essentially done without having to write any code. Nice. Now lets implement the first API call that Appsecute will make to the connector – the listComponents webservice.
As I mentioned above we’re going to let repositories in TravisCi be represented as components within Appsecute, to do this we need to fetch the current users repositories and format them in the way that Appsecute expects. The following code exchanges the GitHub access token that Appsecute hands us in the webservice call for a TravisCi one, it then gets information about the user and finally gets the users repositories and formats them for Appsecute:
If you’re unfamiliar with Node.js this code may look a little foreign, but the concept is fairly simple and can be implemented in any programming language.
We have two more webservices to implement, mapComponent and unmapComponent – Appsecute will call these when users map/unmap components to their applications. These webservices are the ideal place to subscribe and unsubscribe to and from events in the system you’re integrating with. As an example, in the GitHub connector we use these calls to create or delete webhooks. Although TravisCi supports webhooks it doesn’t have an API for creating and deleting them, instead they are defined in a travis.yml file that must be checked in with the projects source code. So we will instead use these webservice calls as an opportunity to generate unique webhook urls that the user can copy and paste in to their travis.yml file:
Looking at mapComponent first, we verify the validity of the access token, generate a unique url that webhooks can be sent to and finally persist it in the database. Notice how we return a response to Appsecute that includes the webhook url? Appsecute will instruct the user on what to do with this:
Our unmapComponent implementation undoes the above code, by deleting the mapping that we persisted it will invalidate the unique url we generated:
That’s it for the webservices we needed to implement for Appsecute, the last thing to do is to create a webservice that TravisCi can call to send webhooks to, this webservice will translate the TravisCi webhook in to an Appsecute event and send it to Appsecute. Something important to remember when receiving webhooks is to make sure it really is the service you’re integrating with that is sending them, and not a malicious user. In the mapComponent method above you will have noticed we generated and embedded a secret inside each webhook url, we will validate this secret when receiving webhooks so we can have some confidence in where they came from. The following code shows the secret validation:
The final code we need to write here is the translation from TravisCi webhook to Appsecute event. This code needs a little bit of thought to make sure we get the desired behaviour when the events are rendered on the Appsecute timeline. If you take a quick look at the event publishing documentation you will see there are a few properties we can tweak to change the event behaviour. The desired result that we want in this specific situation is as follows:
If the build fails, where the previous build was successful, create a new event with the status ‘outstanding’
If the build fails, where the previous build also failed, comment on the existing event (that would have been created from the condition above)
If the build succeeds, where the previous build failed, comment on and clear the existing event (again the one that would have been created from condition 1.)
If the build succeeds, where the previous build was successful, create a new event with the status ‘info’
These sorts of considerations are only needed when building connectors for services that have a state that should be represented in the Appsecute dashboard. Appsecute counts the number of outstanding events for a component and uses it to alert users to issues with the services they use. If you think of a connector like the GitHub one you can see it doesn’t need this type of logic, it can simply raise an ‘info’ level event every time someone commits, there is no state to relay.
I’m not going to show all the code required for this logic as it’s specific to TravisCi but here we can see the webhook web service that TravisCi calls with build results, we parse the webhook body and translate it in to an Appsecute event, finally publishing it to Appsecute:
With all the code written the last thing to do is register the connector with Appsecute, head over to https://stream.appsecute.com/#settings and click the register connector button and fill in the blanks – once your connector is registered you will be given the secret that must be specified when publishing events:
And the result is that we have a shiny new connector for Appsecute users to use:
So there we have it, anyone can build connectors for Appsecute and we’re always standing by to help with any problems or answer questions, so why not build your own connectors and share them?
Please feel free to comment and ask questions or email me directly at [email protected]
Today we opened the doors to the public beta of the new Appsecute – a broader and more collaborative extension of the “old” Appsecute, we’ve gone beyond focusing exclusively on Platform as a Service and extended our single pane of glass to the cloud in general.
The new Appsecute pairs a social stream with a dashboard of application health to create the ultimate real-time application management tool for developers and operations. The social stream shows events from all the services you use (such as GitHub, Zendesk and PagerDuty etc.) and allows you to comment and coordinate work directly on the stream – think of it like a specialized Facebook-style timeline.
We’re excited by the possibilites this way of working enables, the feedback from private beta users has been that Appsecute quickly became the one tool everyone had up on their screen all day to stay on top of things in real-time.
Expect some exciting announcements over the coming weeks as we rapidly iterate and improve the existing features while adding support for new services (AWS, New Relic and TravisCI are in our sites).
Check out the new Appsecute: https://appsecute.com
Today we’re really excited to outline a new vision for Appsecute – it’s social, it’s open, it’s collaborative and it’s extremely addictive!
Ever had that problem where you’re using a dozen different services to manage your application? And you have to log in to them and watch them every day just to stay on top of things? You know what I mean… GitHub for source control, Zendesk for support, Heroku for hosting, TeamCity for continuous integration, Loggly for logs, email threads for discussions, something else for instant messaging… the list goes on.
Let me introduce the solution to this problem – the new Appsecute, the social stream for developers and operations:
Appsecute started out giving you a single place to manage all your PaaS applications, it aggregated info from all the different platforms you might be using and gave you a real time view of application health. It had the concept of organizations and teams, and had a fine grained access control layer to protect production applications.
Now, we’re going beyond PaaS, we’re bringing the single pane of glass and single place of management to the cloud in general – Appsecute is becoming the place you go to every day to stay on top of everything.
Launching in a few weeks, the new Appsecute is a social stream that understands your systems and the software components that make up those systems.
It knows that your system includes some repositories on GitHub, it knows you’re using CircleCi for continuous integration, it knows the production version of your app is running on Cloud Foundry or Heroku and it knows you’re using Zendesk for support. In fact, through our API, it knows about everything that you as DevOps care about.
The real-time stream shows events that are occurring in the services you use and encourages collaboration around those events. So what is an event? An event can be anything – someone pushing code to GitHub, a build failing, tests failing, an application going offline, a support request coming in, a daily summary of the metrics you care about – literally anything. And you and your team can write comments directly against any event, kind of like Facebook for DevOps. It’s a great way to agree what to do about issues that come up, and who is going to take action.
Events are smart, too. We can use them to alert you to issues and tell you once they’ve been resolved. In the screenshot above, to the right of the timeline, you can see all the components you’ve associated with your system – and we can let you know if they’re healthy or not. If a build fails or monitoring detects an issue with a production application, that component will go red until the problem is resolved.
We have an open API that allows anyone to plug a new system in to the social stream – ‘connectors’ can be written and published on Appsecute so that others can use them. And if anyone in the community writes a connector and open sources it we will host it for you!
We’re launching with some connectors we’ve written ourselves – GitHub, Zendesk, CircleCi, Cloud Foundry, AppFog and Heroku – and we will be working hard to keep a constant stream of new connectors coming out the door after launch.
Right now we’re running a private beta of the new Appsecute. If you’re interested in getting in the beta please comment or email me at [email protected]