Custom AWS Cloudwatch metrics


#1

Hi All,

I have a set of AWS Cloudwatch custom metrics that I would like to create some graphs for. Through the current AWS Cloudwatch integration only a certain set of metrics are being pulled in (e.g. no metrics for S3 or Redshift). Currently there is no view on when this will be opened up further. Does anyone have any custom widget that helps me bypass this problem please? Any help would be greatly appreciated!


#2

Hi @bart.schelfhout :wave:

Currently our AWS Cloudwatch integration does not support custom metrics, but I’m happy to add a request for future consideration on your behalf.

If you have a developer available on your team (or are a developer yourself) you could use our Datasets API to display these custom metrics. You would need to write a script/application that authenticates and connects to the Cloudwatch API and requests the data. You can then ‘push’ a dataset that includes the data you would like to display.

You can find more details on using Datasets here: https://www.geckoboard.com/datasets/.


#3

Thanks for your reply Heather.

I do find it a bit peculiar that when GeckoBoard decided to integrate with Cloudwatch, they only selected a handful of the services’ metrics (and not even e.g. the AWS Cloudwatch standard metrics for Redshift for instance.
As an engineer, I find it hard to believe there is a whole lot of code involved with opening up the AWS Cloudwatch plugin to all Cloudwatch based metrics and I would hope there would be quite a lot of support for doing so.

The main thing tearing me away from implementing this myself is that I need to create a new data set for each new metric, and if you have quite a few like myself, that becomes a massive blocker.

Thanks for your suggestion and getting back to me, and I do hope the message reaches the Geckoboard product team that they are missing a trick there. (I suggest updating your marketing documents and stating that you have CLoudwatch integration for a select set of metrics).


#4

Hi Bart, I just wanted to follow up for Heather here. First of all, I understand your interest in Geckoboard offering these metrics – we’d be really glad to make this easier for you! In regard to the work required for us to do that, the biggest blocker is finding the developer bandwidth while we’re focused on many other projects, though I hope my team is able to address this integration specifically in the next few months. Thanks again for your feedback and suggestions!


#5

Hi Jason, thanks for your reply and I appreciate you acknowledging this incompleteness in terms of Cloudwatch integration. Being an engineer myself, I just wonder how the current selection is made in terms of which namespaces you include for Cloudwatch integration with Geckoboard? I understand you are short on developer resources. Is there any way that integration code pieces can be open sourced so we can contribute to it?

Please keep me posted on any timescales you may come about as this road block is forcing us to look around for alternatives.


#6

Hi Bart, yes, I’ll be sure to keep you posted about any updates. The namespaces we currently offer in the integration are mostly historical, just in that those the ones popular/important at the time. I’m not sure that open sourcing the code would be of benefit, since it fits within the framework of the larger Geckoboard app, and thus isn’t very useful alone. But this is something that we’ve thought about in the past, and still may be worthwhile.


#7

Hi there Jason,

Is there any continuation from this at the moment? You mentioned before (in June) that you were hoping to get this onto the roadmap in the coming months, so wanted to check if this has made it onto any roadmap at all.
As our business is using more and more of the AWS available components, only having metrics available for a subset of this is proving to be a bigger and bigger blocker.

Regards


#8

Hi Bart, thanks for following up about this, and sorry for my delayed reply (I was away last week). I’m afraid we haven’t been able to prioritize this :cry: and I’m not sure yet when we would. While not ideal, I’d be happy to give you a hand if you’d like to use our Datasets platform to get this data into Geckoboard. And of course, once I know more about our native CloudWatch integration I’ll be sure to let you know.


#9

Hi Jason,

Thanks for the heads up. I appreciate you coming back to me and I would be keen for you to show me how I can get the sizeable amount of custom and other non custom (but unsupported namespace such as Redshift) cloudwatch metrics into geckoboard. Would your solution aid me in regularly pushing metrics into geckoboard or how would that work. As my need is urgent I would like to tackle this asap.


#10

Hi Bart,

A bit of good news. A colleague of mine made some time to add S3 and Redshift to our prebuilt CloudWatch integration:

I know that it doesn’t completely solve your troubles here, since you still need custom namespaces, but I do hope it’s helpful. Are there any other namespaces besides S3 and Redshift that you want to see available?

As for getting your custom namespace data into Geckoboard, the biggest blocker is that it’s trickier with our current configuration model. We’d still be glad to add it, it’s just not as simple as the other namespaces. So if you’d still be up for a custom approach for this, you’ll need to have an app that connects to CloudWatch to pull the data you want from it, transform the data to a Geckoboard Dataset, and then push it to Geckoboard. There are some examples I’ve seen of using AWS Lambda to connect to CloudWatch, as one approach. If you’re still keen to move forward with that in mind, we can identify 1. where you’d want to host this middleware 2. what language + frameworks you’d like to use for development and 3. what data you want from CloudWatch


#11

Hi Jason,

Thanks for adding those standard namespaces. Could I cheekily request the kinesis namespaces (firehose and streaming) and the Lambda namespace too? Thanks.
In terms of the custom ones; my thought are to have a regular lambda running (say every 15min) that just pulls from cloudwatch and pushes to geckoboard API. The only painful part currently is just to create each dataset manually in geckboard prior to doing so, and I have quite a few metrics that I want to push. any suggestions would be appreciated.

totally unrelated; is it possible to combine 2 cloudwatch metrics into 1 single line chart? I would assume you can but can’t easily find this. Thanks


#12

Hey Bart,

I’m glad that adding those helped. I’ll check in with my colleague if they’re available to add the others, as well.

Sounds like a good plan for he custom namespaces! As for creating the datasets, there may be a way you could programmatically generate them – at least I could imagine generating the dataset name based off of the metric name. And depending on the data you’re getting back, it might be possible to generate the schema. E.g., if you get a JSON array of objects, you could type check the object’s property values and define those as the schema fields.

If you aren’t able to do it that way, you could at least define an array of dataset schemas in your language of choice, and then map over the array and push them up to Geckoboard.

E.g. in Node.js I might do something like:

const gb = require('tuko')('API key here');

const datasets = [
  {
    id: 'metric1',
    schema: { 
      fields: {
        count: { type: 'number' },
        date: { type: 'datetime' },
      },
    },
  },
  { 
    id: 'metric2',
    schema: { 
      fields: {
        count: { type: 'number' },
        date: { type: 'datetime' },
      },
    },
  },
  // etc.
];

Promise.all(datasets.map(dataset => gb.create(dataset.id, dataset.schema)));

Where tuko is a Promise based Node module for Geckoboard’s datasets (that I made, since our official module is still only callback based) – https://www.npmjs.com/package/tuko

Finally in regard to combining two CloudWatch metrics into a single chart, this would only be possible at the moment when you process the data, before you send it to a dataset.

We are however developing a new platform that will in the future let you do this more arbitrarily from within the visualization builder.


#13

Hi @bart.schelfhout! I realize it’s been quite a while since we last talked about this, but we did finally revamp our CloudWatch integration to support custom namespaces and metrics (along with adding all other AWS namespaces!) :tada:

The widgets you have currently on your dashboards won’t have the custom namespaces/metrics available, but if you go to Add widget and find the CloudWatch integration, then all of those widgets will have the custom namespaces and custom metrics available.

If you give this a try please do let me know if you have any feedback to share!