Release Roundup 2017 Q3

Product Updates

Product development has been going full tilt for quite some time now. As a result – we have made some changes to product management – to better meet customer needs (blah blah blah) — you have heard that before.

But it’s true!

We have made internal process changes and we have released more updates, integrations, and use cases in the last 6 months – than in the previous 2 years.

And – we have a lot more coming.

If you want to be a part of this, join our Customer Engagement Group and get involved!

The big driver in our acceleration is a system we call kinops. You can check it out at https://kinops.io … it is our Kinetic platform – pre-installed, hosted on Amazon and ready to use. Our goal is to allow companies of 100-1,000 people to quickly get up and running and benefiting from our Kinetic platform. (Get your own free kinops space here.)

One thing learned from kinops is that when systems get massive – logging gets harder. We think we fixed that with our log system changes. Go read the post here.

Oh yeah – and we have updated a number of integrations:

  • Chef
  • Salesforce
  • Puppet
  • ServiceNow
  • ElasticSearch
  • SOLR
  • GitLab
  • AmazonS3

As always, you can find all of those on http://community.kineticdata.com

Front-ends

  • We have released another front-end bundle to give another choice on how companies choose to manage and optimize their web rendering. This one uses the Facebook React system to give greater performance and flexibility in generating front-ends for the Kinetic platform.
  • We’ve developed a training course on Community.

Next quarter we will be releasing:

  • Datastore
    • Optimized for size and speed, the ability to manage massive (millions, 10s of millions 100s of millions (or way more) assets, people, or whatever you need. More to come on our blog make sure you subscribe in the right-hand margin -> 
  • Queue
    • Optimized for rapid working of assigned tasks. Queue gives the ability for teams to rip through big lists of tasks efficiently, effectively with an eye for automation. More to come.
  • Response aka Discussions
    • People want real-time chat (and more). When we first announced Kinetic Response (the ability for teams to work on crisis issues together), the response was overwhelming. Sadly – we had to back-burner it due to Kinetic Request getting the majority of our attention. Well; Response is back and better than ever imagined. We are rolling it into Request and making it a fundamental piece of our Kinetic platform. More to come.

Lots of great things are happening and it will be easier to share that with you as a result of our product development changes.

Click here if you’d like to join our Customer Engagement Group. A small group of people we talk to regularly to keep building cutting edge products.

A Structured Logging Success Story

How Kinetic Data implemented structured logging to drastically improve application support.

What is the new structured logging feature in the Kinetic Data products?

Starting with Kinetic Task v4.3, Kinetic Core v2.0.2, Bridgehub v1.2, and Filehub v1.2 a new feature was implemented in these applications referred to as ‘structured logging’. Structured logging essentially is ensuring that a log file is a preset, consistent, and machine readable format.

In our most recent releases we provide the following new structured log files:

  • structured.access.log
    • Log entry for every time a resource is accessed through the kinetic application. Who, when, how long, what, etc. This can be used for analytics.
  • structured.application.log
    • Contains entries for application warnings, errors, debug, or trace level messages. Used for troubleshooting.
  • structured.authentication.log
    • Authentication attempts get logged to this file. Who, when, and authentication type. Used for troubleshooting and auditing.
  • structured.system.log
    • Heartbeat checks, application startups and shutdowns, and other non-frequent events like that are the purpose for this log file.

Okay… What is the benefit?

Well, because these logs are machine friendly they can then be digested by Filebeat and thus Elasticsearch. Now the log files can provide more functions than diagnostics or troubleshooting, they can also provide analytics and monitoring as well.

  • Diagnostics and Troubleshooting
    • Fast full text searching on error messages
    • Searching by date and time ranges for errors
    • Find all users who experienced a specific error
    • Find any errors a specific user encountered
    • Check the average response times for a specific user over time. See exactly when things started to get slower for that user.
    • Check for any errors in any application in one search: Core, Task, Filehub, Bridgehub. All at once.
  • Monitoring and Alerting
    • Setup an alert for whenever there are more than 100 failed login attempt in a minute.
    • Alert on API calls taking longer than one second.
    • Monitor for anytime webhooks fail to execute.
    • Monitor current system usage per application.
  • Analytics, Auditing, and Reporting
    • Put together a dynamic graph showing the average response times for requests.
    • Trend on how many errors occur per hour. Little hint, you’ll see a bell curve most likely.
    • Quickly throw together a pie chart showing the top ten active users.
    • Find out the last time a specific user tried to login.

How did we use this to our benefit?

Here are a few things that we’ve already used Elasticsearch, Filebeat, and Kibana for with our structured logs:

  • Discovered numerous HEAD HTTP requests resulting in 405 HTTP responses
  • When John reported a kinops page showing a 500 error, the log entry for the error was quickly retrieved despite not knowing exactly when the error occurred. We were simply able to query for: app.user:john.sundberg@kineticdata.com AND level:(WARN ERROR)
  • Developed a ‘kinops Health’ Dashboard to show average request response times, errors counts, and access volume by spaces and users.
  • Setup alerting for when webhooks fail.
  • Helped improve our structured logging before release by using Filebeat and Elasticsearch.
    • Found multiple log entries that did not match our expected patterns which forced us to clean up our logging in the application.
    • When running a report to find top active IP addresses on kinops.io, we did not find nearly as many as we were expecting. This led us to realize we forgot to add support for X-Forwarded-For to get user IP addresses anytime a proxy is used.

Words, words, words. Show me something cool.

The following screenshots of graphs are all from Kibana

Dashboard Screenshot 1
Image shows HTTP response codes and Request times.

 

Dashboard Screenshot 2
This was a graph put together in less time than it took to listen to Waterfalls by TLC. The purpose of creating it was to illustrate how easy it is to get a feel for a user’s experience overtime.