PUT data in dataset with a loop


#1

Hello,
I’m trying to PUT datas in a Dataset with a loop in order to replace datas but I do not know how to realize it.

Is someone have an idea?

Here is my code :

gb.datasets.findOrCreate(
  {
    id: 'opsgenie.listalerts',
    fields: {
      tinyid: {
        type: 'string',
        name: 'TinyId'
      },
      alias: {
        type: 'string',
        name: 'Alias'
      },
      message: {
        type: 'string',
        name: 'Message'
      },
      status: {
        type: 'string',
        name: 'Status'
      }
    }
  },
  function (err, dataset) {
    if (err) {
      console.error(err);
      return;
    }
    for (var i = 0; i < nb; i++) {
    
      var Id = string(tabId[i]);
      var Alias = string(tabAlias[i]);
      var Message = string(tabMessage[i]);
      var Status = string(tabStatus[i]);
      dataset.post(
        [
          { tinyid: Id, alias: Alias, message: Message, status: Status }
        ],
        function (err) {
          if (err) {
            console.error(err);
            return;
          }
          console.log('Data N°'+i+' Added.');
        }
      );
    }
  }
);

I have previously recorded them in tables. (ex: tab[Id] contains all IDs)

I have this error :
Error: The response from the server was not valid JSON

Thank you


#2

Hi,

I’ve been looking into this for you and I’ve had some success but I have not found a way to do exactly what I think you’re looking for.

I did manage to create some code that works reliably to PUT into a Dataset via a loop but this restricts the Dataset to a single item at a time - the last item in the loop.

gb.datasets.findOrCreate(
  {
    id: 'opsgenie.listalerts',
    fields: {
      tinyid: {
        type: 'string',
        name: 'TinyId'
      },
      alias: {
        type: 'string',
        name: 'Alias'
      },
      message: {
        type: 'string',
        name: 'Message'
      },
      status: {
        type: 'string',
        name: 'Status'
      }
    }
  },
  function (err, dataset) {
    if (err) {
      console.error(err);
      return;
    }

      for (var i = 0; i < nb; i++) {
      var Id = tabId[i];
      var Alias = tabAlias[i];
      var Message = tabMessage[i];
      var Status = tabStatus[i];
      dataset.put(
        [
          { tinyid: Id.valueOf(), alias: Alias.valueOf(), message: Message.valueOf(), status: Status.valueOf()}
        ],
        function (err) {
          if (err) {
            console.error(err);
            return;
          }
          console.log('Data added');
        }
      );
    }
  }
  );

I also have a method that you can use to loop through and store the values in a single array, then use PUT to add these to the dataset:

function loopthrough() {
        var values = [];
        for (var i = 0; i < nb; i++) {
        var Id = tabId[i];
        var Alias = tabAlias[i];
        var Message = tabMessage[i];
        var Status = tabStatus[i];
        values.push(
          { tinyid: Id.valueOf(), alias: Alias.valueOf(), message: Message.valueOf(), status: Status.valueOf()}
        );
        console.log('Data N°'+i+' Added.');
      }
      return values;
    }

gb.datasets.findOrCreate(
  {
    id: 'loopthrough.tryagain',
    fields: {
      tinyid: {
        type: 'string',
        name: 'TinyId'
      },
      alias: {
        type: 'string',
        name: 'Alias'
      },
      message: {
        type: 'string',
        name: 'Message'
      },
      status: {
        type: 'string',
        name: 'Status'
      }
    }
  },
  function (err, dataset) {
    if (err) {
      console.error(err);
      return;
    }
      dataset.put(
        loopthrough(),
        function (err) {
          if (err) {
            console.error(err);
            return;
          }
          console.log('Data Added.');
        }
      );
    }
  );

I have used the console to log the individual data items being added to the Array as it is not possible to log them being added to the dataset in this way.

In order to loop through and add multiple items individually, you would need to use POST, to append these. The above code does not work with POST and I have not been able to find a way to do this reliably.

@jason might have a few more thoughts! :slight_smile:

Lisa


#3

Hi @AntoineMOREL, can you explain more why you want to do it in a loop? Also you say that you want to use PUT, but then in your code you use dataset.post. Also, on these lines:

      var Id = string(tabId[i]);
      var Alias = string(tabAlias[i]);
      var Message = string(tabMessage[i]);
      var Status = string(tabStatus[i]);

string is not a function. It must be uppercase, i.e. String in order to make those values into strings. Unless you have it defined elsewhere in your code?

Also the error you’re seeing is a false flag. Since you’re calling datasets.post, it requires 3 arguments, but when only 2 are supplied, the callback is undefined. I.e. it should be dataset.post(data, options, callback). The error is

cb(new Error('The response from the server was not valid JSON'));
^
TypeError: cb is not a function

I.e. the actual error is that there is no callback supplied. If you change the .post method to .put, it works.

Also most problematic is using the for loop to do this. Once you fix the above errors, you’ll actually see:

Data N°3 Added.
Data N°3 Added.
Data N°3 Added.

(or instead of 3 whatever nb is equal to). This is because of how closures work in JS. In a sense, only once all of the iterations have passed through in the for loop will the callbacks be called, when i is bound to the last value it was assigned in the for loop.

This also means that only one value (the last value, as @GeckoLisa found) will be pushed to the dataset.

Luckily there are some helpful ways JS provides to do this, namely Promise and .map or .forEach.

With Promise and .map:

  function (err, dataset) {
    if (err) {
      console.error(err);
      return;
    }
    Promise.all(tabId.map(function (i) {
      var Id = String(i);
      var Alias = String(tabAlias[i]);
      var Message = String(tabMessage[i]);
      var Status = String(tabStatus[i]);

      dataset.put(
        [
          { tinyid: Id, alias: Alias, message: Message, status: Status }
        ],
        function (err) {
          if (err) {
            console.error(err);
            return;
          }
          console.log('Data N°'+i+' Added.');
        }
      );
    }));
  }

The order is not sequential, so e.g. I see for 3 items with indexes 0, 1, 2 output of:

Data N°1 Added.
Data N°2 Added.
Data N°0 Added.

Or with .forEach:

  function (err, dataset) {
    if (err) {
      console.error(err);
      return;
    }
    tabId.forEach(function (i) {
      var Id = String(i);
      var Alias = String(tabAlias[i]);
      var Message = String(tabMessage[i]);
      var Status = String(tabStatus[i]);

      dataset.put(
        [
          { tinyid: Id, alias: Alias, message: Message, status: Status }
        ],
        function (err) {
          if (err) {
            console.error(err);
            return;
          }
          console.log('Data N°'+i+' Added.');
        }
      );
    });
  }

I hope that helps! You’ll want to look into JS closures, e.g. here – https://stackoverflow.com/questions/3572480/please-explain-the-use-of-javascript-closures-in-loops and Promises and the iterator methods .forEach and .map.


#4

Many thanks to @GeckoLisa and @jason for your help.

Here is my new code :
//OpsGénie
var opsgenie = require(‘opsgenie-sdk’);

var nb = 0;   //nombre d'alertes
var monJson;  //récupère les données
var tabId = [];   //récupère les Id
var tabAlias = [];   //récupère les Alias
var tabMessage = [];   //récupère les Message
var tabStatus = [];   //récupère les Status

opsgenie.configure({
'api_key': ''
});

var list_alert_json = {
query : "status : open",
offset : 0,
limit : 3,
sort : "alias",
order : "desc"
};

opsgenie.alertV2.list(list_alert_json, function (error, alerts) {
if (error) {
    console.error(error);
}
else {

    //console.log("List Alert Response");
    //console.log(alerts);

    monJson = alerts;   //monJson récupère les données
    nb = Object.keys(monJson['data']).length;   //nb = nombre d'alertes
    console.log();
    console.log("Nombre d'alerts = " + nb);
    console.log();

    //Longueur tableau = nombre d'alertes
    tabId.length = nb;
    tabAlias.length = nb;
    tabMessage.length =nb;
    tabStatus.length =nb;
    console.log();
    console.log("Tableau = " + tabId.length);

    for (var i = 0; i < nb; i++) {
        tabId[i] = String(monJson['data'][i]['tinyId']);//Insert les id dans TabId
        tabAlias[i] = String(monJson['data'][i]['alias']);//Insert les alias dans TabAlias
        tabMessage[i] = String(monJson['data'][i]['message']);//Insert les messages dans TabMessage
        tabStatus[i] = String(monJson['data'][i]['status']);//Insert les status dans TabStatus

        var Id = tabId[i];
        var Alias = tabAlias[i];
        var Message = tabMessage[i];
        var Status = tabStatus[i];

        //Affiche les données récupérées sous format Json
        console.log(
            [
              { tinyid: Id.valueOf(), alias: Alias.valueOf(), message: Message.valueOf(), status: Status.valueOf()}
            ]
        );
        console.log();
    }

}
});

//Geckoboard
var API_KEY = '';

var gb = require('geckoboard')(
API_KEY
);

gb.datasets.findOrCreate(   //Cherche ou créé le Dataset
  {
id: 'postdataset.loop',   //Id du Dataset
fields: {   //Champs du dataset
  tinyid: {
    type: 'string',
    name: 'TinyId'
  },
  alias: {
    type: 'string',
    name: 'Alias'
  },
  message: {
    type: 'string',
    name: 'Message'
  },
  status: {
    type: 'string',
    name: 'Status'
  }
}
  },

  function (err, dataset) {
if (err) {
  console.error(err);
  return;
}

for (var i = 0; i < nb; i++) {

  var Id = tabId[i];
  var Alias = tabAlias[i];
  var Message = tabMessage[i];
  var Status = tabStatus[i];

  dataset.post(
    [
      { tinyid: Id.valueOf(), alias: Alias.valueOf(), message: Message.valueOf(), status: Status.valueOf()}
    ],
    null,

    function (err) {
      if (err) {
        console.error(err);
        return;
      }
      console.log('Data Added.');
    }
  );
}
  }
);

the strange thing is that:
Whatever I choose as number of alerts to draw (limit), I have as error “Error: Unknown error” for the first half and “Error: The response of the server was not valid JSON” for the second
Example 1
Limit to 10
-> first half: “Error: Unknown error”, the second half: "Error: The response from the server was not valid JSON"
Example 2
Limit to 3
-> The first two: “Error: Unknown error”, the last one: "Error: The response from the server was not valid JSON"
I find it disturbing …: /


[HELP] Not valid Json for Brand24 Integration
#5

Hey @AntoineMOREL,

This is because in JavaScript and Node, you can’t exactly write “linear” code. In the first part of your code, you send a request to get the data from OpsGenie.

opsgenie.alertV2.list(list_alert_json, function (error, alerts) {

but because of how JS and Node work, the code then immediately goes through to this line:

gb.datasets.findOrCreate(

So this means that the data that you’re trying to send to Geckoboard is possibly empty, because it may not yet have been populated in the callback to the OpsGenie call.

What I mean is that the callback for opsgenie.alertV2.list` isn’t guaranteed to run before you create and send data to your dataset.

The solution is that you need to put your dataset creation and update inside the callback of the OpsGenie call, like this:

var nb = 0;   //nombre d'alertes
var monJson;  //récupère les données
var tabId = [];   //récupère les Id
var tabAlias = [];   //récupère les Alias
var tabMessage = [];   //récupère les Message
var tabStatus = [];   //récupère les Status

opsgenie.configure({
'api_key': ''
});

var list_alert_json = {
  query : "status : open",
  offset : 0,
  limit : 3,
  sort : "alias",
  order : "desc"
};

opsgenie.alertV2.list(list_alert_json, function (error, alerts) {
  if (error) {
    console.error(error);
  }
  else {

    //console.log("List Alert Response");
    //console.log(alerts);

    monJson = alerts;   //monJson récupère les données
    nb = Object.keys(monJson['data']).length;   //nb = nombre d'alertes
    console.log();
    console.log("Nombre d'alerts = " + nb);
    console.log();

    //Longueur tableau = nombre d'alertes
    tabId.length = nb;
    tabAlias.length = nb;
    tabMessage.length =nb;
    tabStatus.length =nb;
    console.log();
    console.log("Tableau = " + tabId.length);

    for (var i = 0; i < nb; i++) {
      tabId[i] = String(monJson['data'][i]['tinyId']);//Insert les id dans TabId
      tabAlias[i] = String(monJson['data'][i]['alias']);//Insert les alias dans TabAlias
      tabMessage[i] = String(monJson['data'][i]['message']);//Insert les messages dans TabMessage
      tabStatus[i] = String(monJson['data'][i]['status']);//Insert les status dans TabStatus

      var Id = tabId[i];
      var Alias = tabAlias[i];
      var Message = tabMessage[i];
      var Status = tabStatus[i];

      //Affiche les données récupérées sous format Json
      console.log(
        [
          { tinyid: Id.valueOf(), alias: Alias.valueOf(), message: Message.valueOf(), status: Status.valueOf()}
        ]
      );
      console.log();

      //Geckoboard
      var API_KEY = '';

      var gb = require('geckoboard')(
        API_KEY
      );

      gb.datasets.findOrCreate(   //Cherche ou créé le Dataset
        {
          id: 'postdataset.loop',   //Id du Dataset
          fields: {   //Champs du dataset
            tinyid: {
              type: 'string',
              name: 'TinyId'
            },
            alias: {
              type: 'string',
              name: 'Alias'
            },
            message: {
              type: 'string',
              name: 'Message'
            },
            status: {
              type: 'string',
              name: 'Status'
            }
          }
        },

        function (err, dataset) {
          if (err) {
            console.error(err);
            return;
          }

          for (var i = 0; i < nb; i++) {

            var Id = tabId[i];
            var Alias = tabAlias[i];
            var Message = tabMessage[i];
            var Status = tabStatus[i];

            dataset.post(
              [
                { tinyid: Id.valueOf(), alias: Alias.valueOf(), message: Message.valueOf(), status: Status.valueOf()}
              ],
              null,

              function (err) {
                if (err) {
                  console.error(err);
                  return;
                }
                console.log('Data Added.');
              }
            );
          }
        }
      );
    }

  }
});

I’m still confused why you need to do this in a loop, though. And you’re still going to run into problems with closures in the for loop. Since you have all the rows of data already, why not post them all at once? It really simplifies the code:

opsgenie.configure({
  'api_key': ''
});

var list_alert_json = {
  query : "status : open",
  offset : 0,
  limit : 3,
  sort : "alias",
  order : "desc"
};

var API_KEY = '';
var gb = require('geckoboard')(API_KEY);

opsgenie.alertV2.list(list_alert_json, function (error, alerts) {
  if (error) {
    console.error(error);
    return;
  }
  //console.log("List Alert Response");
  //console.log(alerts);

  //Geckoboard
  gb.datasets.findOrCreate( //Cherche ou créé le Dataset
    {
      id: 'postdataset.loop',   //Id du Dataset
      fields: {   //Champs du dataset
        tinyid: {
          type: 'string',
          name: 'TinyId'
        },
        alias: {
          type: 'string',
          name: 'Alias'
        },
        message: {
          type: 'string',
          name: 'Message'
        },
        status: {
          type: 'string',
          name: 'Status'
        }
      }
    },
    function (err, dataset) {
      if (err) {
        console.error(err);
        return;
      }

      var data = alerts.data.map(function(i) {
        return {
          tinyid: i.tinyId,
          alias: i.alias,
          message: i.message,
          status: i.status,
        };
      });

      dataset.post(data, null, function (err) {
        if (err) {
          console.error(err);
          return;
        }
        console.log('Data Added.');
      });
    });
});

#6

Hello Jason,

The only problem was the message that is more than 100 characters long
Both for my code and yours :joy:

But your version is just perfect so here is the final version :

//OpsGénie
var opsgenie = require('opsgenie-sdk');

opsgenie.configure({
  'api_key': ''
});

var list_alert_json = {
  query : "status : open",
  offset : 0,
  limit : 10,
  sort : "alias",
  order : "desc"
};

var API_KEY = '';
var gb = require('geckoboard')(API_KEY);

opsgenie.alertV2.list(list_alert_json, function (error, alerts) {
  if (error) {
    console.error(error);
    return;
  }
  //console.log("List Alert Response");
  //console.log(alerts);

  //Geckoboard
  gb.datasets.findOrCreate( //Cherche ou créé le Dataset
    {
      id: 'opsgenie.listalerts',   //Id du Dataset
      fields: {   //Champs du dataset
        tinyid: {
          type: 'string',
          name: 'TinyId'
        },
        alias: {
          type: 'string',
          name: 'Alias'
        },
        message: {
          type: 'string',
          name: 'Message'
        },
        priority: {
          type: 'string',
          name: 'Priority'
        },
        status: {
          type: 'string',
          name: 'Status'
        }
      }
    },
    function (err, dataset) {
      if (err) {
        console.error(err);
        return;
      }

      var data = alerts.data.map(function(i) {
        return {
          tinyid: i.tinyId,
          alias: i.alias,
          message: i.message.slice(0, 99),
          priority: i.priority,
          status: i.status,
        };
      });

      dataset.put(data, function (err) {
        if (err) {
          console.error(err);
          console.log(data);
          return;
        }
        console.log('Data Added.');
      });
    });
});

#7

Yo can automatically update the Dataset by putting the code below inside this :

setInterval(Looper, 300000)//time (300000 ms = 5min) 

function Looper(){
// the code
}

#8

Hello, i would like to do exactly the same thing for a custom list widget.

I thought to use geckoq module with a loop but the informations are each time replaced by the new datas.

Can anybody help ?

Thanks.


#9

Hi @AntoineMOREL, sorry for my delayed reply here, I’ve been away on holiday.

Can you explain again why you need to use a loop? There is no way around the data being replaced each time you push a “Custom widget”, that’s why we created the Datasets platform. But if you share overall what you’re trying to achieve, maybe it’s possible in a different way.