Salesforce Metadata Management (node.js) - Part 1

Introduction

I blogged a few weeks ago about my love for Node and introduced briefly one of the app I did on it. Time has come to speak a bit more about it. When usually people want to discover a new language, they make a hello world. Even advanced developer like to write hello world blog post, no offence to this guy though. However, if everybody starts writing a hello world snippet, that's not really much exciting after 2 hours, you want something more! But well, I started on Node with a hello world, and this tutorial helped me a lot or this one if you understand French, it's even better explained than the original one in English :-) Don't worry, I won't speak longer now about hello world!

Project

Now that I had a target, i.e. writing a complex program, I still had to find what does not exist on the market and what I'd like to achieve. One week before, I had overwritten by mistake metadata on a UAT Sandbox. No drama, but that's never cool, especially when you have to find someone who has eventually a backup of the source code and pray that this guy has still the code. Thank to this shameful event, I had my project! Writing an app which backup Salesforce metadata into GIT.

Start and Proof of Concept

I started to implement the Salesforce REST API by using custom SOQL on the Class/Trigger/Components/Pages Object. I didn't want to spend time with the Metadata API, which is very inconvenient to use (SOAP...). Once the authentication was fine, I have choosen to backup data on GitHub, because they have probably the most complete GIT API (then come most likely Bitbucket). I know very well the Salesforce API so this was quite straightforward and the GitHub API was no big mistery. The app was working rather well after a few days. However, it came up different problems. First one was that backing up just the 4 Apex / Page Objects is not much, second GitHub is cool but if you want to have private repo, it's expensive, well it has an high price to be politically correct. And it's usually not an option to store a full client Org into a public GitHub repo!

Bitbucket and their weak API

A colleague told me the awesomeness of Bitbucket and that I should have a look at it, thing that I did. If Bitbucket is a 95% copy of GitHub, their API is unfortunately not that great. However, you can have as many private repo as you want, it's totally free. So I tried to use their API the way I could but struggle to use it successfully and eventually gave up because there was no option. So what the point of using Bitbucket if I can't use their REST API!?

Heroku Command Line

Luckily you can use in Heroku the command line (through your program written in node, ruby, ...). For Node, it's the exec command or execFile if you want to run a binary directly! Second awesome thing, was that Heroku knows already the git command! All I need to write was a git init, git add ., git commit -m "my commit", git push bitbucket master. Oh wait! git push bitbucket master is cool on your desktop computer, but what about Heroku? Because obviously "bitbucket" is not known. At this point, I started to have headaches but I never give up! I tried to replace the "bitbucket" with its normal name which is git@myKey:gitUsername/myFolder.git. Better, but we run into our next problem. How does Heroku know, what is myKey? Being not a GIT expert, I had to dig into the very user-friendly GIT documentation and browse the web to find a solution to my problem. Also, I had to deal with different ssh key, because of different users. Each user has most likely its own key. Finally I found out that Heroku allows the creation of files because Cedar (the name of the stack where node.js app are working on Heroku) has an Ephemeral Filesystem. The light at the end of the tunnel? Almost! Being able to write file in Heroku, I could just replicate how GIT is working on a normal computer. Finally, I ended up creating on the fly ssh Key under the .ssh/ directory and creating a config File called config also in the .ssh/ directory. In this file, I give the Host (which is the reference of myKey), the Hostname (most likely bitbucket.org or github.com) and the Identity file, which is the reference to the .ssh key file! I had of course to bulkify that if I have different users using the app on the same dyno. Cool, isn't it? All this complicate stuff, just to use Bitbucket over GitHub, because we are poor guy who can't afford private repo of GitHub.

Full Salesforce Metadata

Half the way done. I spent weeks to find out how to backup stuff on Bitbucket, I can't backup now just classes, triggers, pages and components; I have to backup all everything! If this step was less mistery than the git command line, this was still not easy. Node is still a quite new language and I didn't find really a tool, which has convinced me on how to parse a WSDL. Also, Salesforce WSDL is quite complicate and I finally preferred doing the parsing by myself. At the end, it was also less ugly that what I thought and I could create small part of SOAP, which I reused where I could. I won't enter in the detail here, it's quite boring and straightforward work.

A few more details

Let's summarise: I pull metadata from Salesforce, I push them to a bitbucket or github server, what happens in the middle, on the Heroku server? Yes because if I use GIT, I have to save the metadata temporary on the Heroku Server! As I explained before, it's possible to write files on the Cedar Stack. However, there is no zip extractor, which is running very well in Javascript. The best is still JSZip which has a node.js module called node-zip. However, I experienced some problem when unzipping and I had to find another solution. Unfortunetly, there is no unzip command on the Heroku Command Line so I found finally a binary, which could get executed with the execFile command. Now, you understand how the app is working.

A last piece to the puzzle

However there is still something missing, a detail which makes the backup failing. Whoever has already used GIT knows this problem: non fast-forward push. Indeed, if you change of working directory and try to push the new working directory into an existing git directory, you will get this kind of error. So what's the solution? On your local machine, you have most likely to pull the remote directory, merge your files, and push that again... On Heroku, I do the same, almost. I first pull the remote directory (extract it with my unzip binary), delete all everything but the .git hidden directory which has now the old informations, then I pull the Salesforce metadata into the working directory, unzip it, make git add . -A in order to remove old files in the git server and push that to Bitbucket or GitHub! Not bad, and it's working :-)

Scheduling

Because I didn't want to spend physically money in an app on which I does not earn a single penny, I also didn't want to scale my app with 2 dynos, one for the worker and another one for the web. So I found a trick, again. When you create an Heroku App, you can create a free Postgres Database. It's where I store User Config, Git Username, Git Private Token, ... You get an Environment Parameter, which is called something like HEROKU_POSTGRESQL_SILVER_URL. No mistery behind, it's just a URL to a small database hosted on AWS which looks like postgres://username:password@ec1.compute-1.amazonaws.com:5432/table. Once you know this URL, you can reuse it in another app - and it's what I did. I finally created 2 free Heroku App, one which is the web front-end, another one which is the worker back-end, both sharing the same database. Also, I didn't find any free Scheduler so I created mine in Javascript:

var bgBackup = require('./routes/bgBackup')
    ,exec = require('child_process').exec;

start();

function start(){
    console.log('start process started...');
    setTimeout(function(){
          start();
    }, 1000 * 60 * 10);
    bgBackup.startBackup(null, null, null);
}

process.on('uncaughtException', function(err) {
  console.log('UNCAUGHTEXCEPTION: ' + err);
  var curlURL = 'curl -H "Accept: application/json" -u :'+process.env.APITOKEN+' -X POST https://api.heroku.com/apps/myworkerapp/ps/restart';
  exec(curlURL, function (error, stdout, stderr) {
      console.log('process restarted!');
      if(error){
          console.log(error);
      }
  });
});

It runs every 10mn (1000*60*10 ms) and of course, if an error is happening, I may have to restart the app. I do for that a simple curl command which use the Heroku API to restart app.

A fully working demo of this app can be found at this address.

comments powered by Disqus