Google+ API broke again - G+7 does not work any more

Google decided to change their internal API yet again (yes, official API is still crap, I cannot use it for the G+7 gadget). This time breakage is pretty substantial and some work will be required to work around it.

I should be able to work around this - I have some rough fixes read now, testing them.

Please give me some time to test and deploy (I should be done in less than a day). 


UPDATE: I have just released version 1.11 of G+7, with support for the new behaviour of Google+
get it from http://www.kalamonsoft.com/gp7

G+7 1.9 - fixes post details display and adding comments

In my last post I complained about Google+ changing their REST API, which broke adding commnents. I did not really have time to fix this, working on some other cool stuff.

But now Google+ broke their API and Javascript some more, which caused G+7 to stop displaying post details in the flyout. Which required some attention from me.

Now I have released version 1.9 1.10 of G+7, which fixes both issues.

It is available here

UPDATE: Turns out that Google changed their Javascript yet again on May 1st. This time they changed some function name for no apparent reason at all, requiring me to fix G+7 again - hence version 1.10. Damn, this is getting a bit boring

Enojy.

Adding comments to Google+ from G+7 is currently broken

It looks like Google changed something in their REST API and they broke G+7's ability to add comments. Oh well. I will need to fix it

Google+ Changed, G+7 Had To Be Fixed

I have noticed a couple of days ago, that very important functionality in the G+7 desktop gadget stopped working. You could no longer view and add comments, add +1s or share.

Some investigation revealed that Google changed they way they handle dynamic content of Google+. It was a pretty nasty change, as the initial data is no longer included in the page content as a static table, but dynamically created in parts, using Javascript. This made my post about parsing Google+ post pages obsolete. As well as broke G+7.

Fortunately, I managed to reverse-engineer what Google does yet again and prepared version 1.8 of G+7 that brings back the missing functionality.

Go here to download it and enjoy.

I am going to prepare a blog post soon, outlining what Google is doing now on their Google+ pages and how you can parse the new format of the data.

Are We Done Yet? Or - Earned Value Calculations in Agile Projects

Don't worry, this is not one of these posts where someone attempts to cram Scrum burn-down charts into waterfall CPI/SPI to satisfy upper management who don't speak agile. This post is going to describe a metric that can be useful to project managers running software projects using agile methodologies. The metric is going to tell you how close to "done" you are in your project. Or more precisely: how close the value delivered by your project so far is to the total expected value of the project.

I have been using agile software development methodologies for a few years now. One thing always surprised me - while agile has effort tracking figured, described and codified as a set of useful practices, tracking earned value for projects is lacking. Up until recently, it was hard to find any mention of agile value tracking on the internet. These days, some authors write about it, but somehow the subject is not very popular in agile teams.

The Confusion

I have witnessed time and again that project teams tend to confuse effort spent (or estimated) with value delivered (or promised). What the popular metrics (like agile burn-down charts or traditional CPI/SPI values) do measure is effort, time and budget spent, and how they compare to the plan. And while very useful, effort metrics are not equivalent to value metrics. Not even close to equivalent to be exact. Why? Quite simple. Sometimes features that are very valuable are not very costly. Sometimes the opposite happens - you are working on something that requires a lot of work, but the return on investment is questionable. In a typical sofware project, a relation of a cost of a feature to its value is vary, very non-linear. Yet, the fallacy of equaling effort spent (or, in other words, sunken cost) to value delivered is very very common.

Why is knowing the value delivered useful? Well, it lets you decide whether it is worthwhile to continue the project or not. In fact, under normal circumstances, this metric should be the metric deciding the fate of your project. It is much more important than budget, time and effort overruns: you will want to continue a project that continues to bring you satisfactory amount of value regardless of whether it costs more than you expected or not. And you will want to cancel a worthless project even if it is cheap, because you can redict your people and resources to more promising undertakings.

Value Points

To measure value delivered by your project as you keep working on it is to add one more parameter to each of your user stories. The metric is called Value Point (VP). Value Point is vary much like a Story Point (used for effort estimation and tracking), in that it is relative. It tells you how the value of a story compares to the value of some other story. Which is useful, because it lets you estimate stories for value using not just concrete financial metrics, but also more fuzzy things (like gut feeling). 

Who assigns the Value Points? The ones responsible for this are project sponsors and any other party that have vested interest in the outcome of the project, can benefit from it, or can provide input for a value of particular story. How is the value points assignment done? the simplest way is to use an equivalent of Scrum release and iteration planning poker and calculate the average of VPs assigned by each participant of the planning meeting. A Fibonacci sequence (1, 2, 3, 5, 8, 13, 21, 34, 55, 89...) is used to establish "legal" values. That is the methodology that is suggested by majority of texts on the subject.

However, other methods can also be used and are sometimes more adequate. For example, there is nothing sacred about using Fibonacci sequence here. You can use a continuum of values, fractional values, span many orders of magnitudes (this is a no-no for Story Point-based effort planning, but for value estimations, it is not uncommon to be able to tell with confidence that something is 200x more valuable then something else in the same project). Best of all, as long as the decision-making team is stable, you don't really need to reach any sort of a consensus and to pick an average Value Points number. Instead, you can just calculate a total out of everybody's assigned Value Points and use this as a final VP number for a story (in contrast to this, in effort planning, a consensus is absolutely required)

What Are Value Points Good For?

After you assigned value points to stories in your project, you are able to provide very useful metrics to the project management team

• is the project still delivering as much value as it did when we first started it? 
• If the value stream dropped, is the value being delivered high enough to justify continuing the project?
• what is the percentage of "very valuable but easy to implement" features? To establish that, you pick stories that have high VP metrics and low Story Point metrics
• what is the percentage of "not very valuable but hard to implement" features? If you have stories like that, you may want to consider dropping them (or maybe postponing until they become more valuable)
• last but not least: how "done" are you in the project - meaning - what percentage of value points has been completed in the project. Upper management very often craves for this sort of a metric

Want to record your desktop? Try DeskBeam Junior

Sometimes you just want to scratch that itch.  For some time, I have been wondering what would it take to write a screen recording application. For Windows. The existing ones were either too primitive, or too expensive. 

All the freeware recorders that I could find recorded either with lossy compression, which made the resulting video stream look awful (not to mention they dropped frames like crazy at higher video rates, because MP4 compression is CPU intensive), or with no compression at all, which made the recorded files huge (like 1GB per minute huge).

Then there were the "it's free, but" variety: free, but limited redording length (varying from 15 seconds to 5 minutes), free but with watermark added to the video, or free but only work with the vendor's own video sharing web service. None of these was good enough for my liking.

And last but not least, there are the expensive recorders with all sorts of bells and whistles. Good products, but costly.

I wanted to have something modest and simple (but not primitive), which would be able to capture desktop activity with 100% fidelity, apply a decent compression, so that the recording has sane size and which woudl allow me to perform some basic editing, such as

• trimming the beginning and the end of the recording
• fading in and out
• title and credit screens
• watermarking
• adding background audio

Plus I wanted to be able to compress the result to WMV (remember, we are talking Windows-based recorder here) and upload it to YouTube for the world to see.

The work took me a good while, because the problem turned out to be more difficult than I thought. After I solved the main obstacles of recording the video, all kinds of details to address popped up (there are always problematic details to address, aren't there?).

But eventually the project was finished. You can download the DeskBeam Junior installer from http://www.kalamonsoft.com/dbjr and try it out.

Or take a look at some screenshots to whet your appetite

Recorder dialog

Player/editor

Compressor

YouTube uploader

Using Google+ API - The Real One - Part 2. Ajax, Paging and Commenting

In the first installment of my blog series about using the real Google+ API, I have described how to retrieve a Javascript structure, representing messages by parsing a Google+ page source. This was a good introduction, buty there is a better way to retrieve a message stream - by calling Google+ server directly using Ajax calls. This method of communicating with Google+ is better than screen scraping described before, because it allows you to get more than the first page of messages

The Message Stream Ajax Call URL

In order to retrieve a message stream from Google+, perform a HTTP GET request to this URL:

https://plus.google.com/_/stream/getactivities/?sp=[1,2,null,circleid,null,20,null,"social.google.com",[]]&rt=j&ct=token

In order to overcome the potential problem with caching (Internet Explorer tends to return cached version of the server response instead of actually contacting the server), you will want to append a dummy timestamp parameter to the request, like this:

'&_requid=' + new Date().getTime()

There are three important parameters of the call, which let you control what gets returned. I have marked these in red color in the URL above. 

• The first one - circleid - is the identifier of the circle, from which you want to retrieve the message stream. If you want to retrieve messages from "all circless", set this to null. Otherwise, you will have to know the ID of the circle you want to retrieve messages from. I will show you how to retrieve information about circles in the next installment of this series
• the second parameter in red is the page size. The default is 20 messages, but you can set this to something else, depending on your needs
• the third parameter - &ct=token - is the "page token" of the page of message stream that you want to retrieve. This is an optional parameter - you should omit it if you want to retrieve the first page. Otherwise, you have to know what the page token of the page that you want to retrieve is. I will describe this in more detail below.

Retrieving the Message Stream

Given the URL above, you can execute the following jQuery to retrieve message stream:

var frontGarbage = "Invalid JSON: )]}'"; 

jQuery.support.cors = true;

 jQuery.ajax({

        type: 'GET',
        url: url,
        error: function (xhr, status, error) {
            if (error.indexOf(frontGarbage) == 0) {
                var response = jQuery.parseJSON(error.substring(frontGarbage.length));
            }
        }
    });

Before we dig into what is the content of response , I first need to explain two things:

• why is the response retrieved in the error handler and not in the success handler of the jQuery Ajax call? 
• what is frontGarbage?

The answer is - the first is the consequence of the second. Google+ server, in all its wisdom, prefixes the proper JSON response of the message stream (and pretty much of any response that it returns to the callers) with the garbage text. This is probably an attempt to make my life more difficult. As a result of this, jQuery is unable to automagically parse the response as JSON and decides that the server errored. Fortunately, it gives us the full server response in the error parameter, allowing us to parse the JSON manually

Now, on to the response data. The majority of it, namely response[0][1][1][0] is the message table containing messages in the exact same format as described in my previous post. You can interpret it in the same way, no modifications required.

There is an additional important parameter in the response: response[0][1][1][1]. It is a pretty long string of hexadecimal characters, which constitutes the "next page token" value for a page that you have just retrieved. Which means that if you pass this value as the ct parameter of the Ajax call's URL, you will get the next page of messages. Curiously, there does not seem to be a "previous page token" anywhere in the response. Which means that you have to remember the already retrieved tokens in some kind of a table, and use the correct one to move back and forth between the message stream pages

Adding Comment To The Post

Now let's concentrate on a stuff that is actually absolutely impossible in the current incarnation of the Google+ API. Which is - let's modify something in the message stream, which the official API does not let you do, because it is a read-only API.

Before we begin - you may be wondering now, why I have described two methods of retrieving a message stream: by screen scraping the HTML page, and by means of an Ajax call. The reason for this is that in order to perform any sort of non-read-only operation, you need to know the XSRF token. Otherwise, Google+ server will reject the call (and rightfully so - we don't want Google+ to be insecure, do we?). For obvious reasons, the XSRF token cannot be obtained by an Ajax call - it has to be contained on a page. So to get to the token required to modify a message on Google+, you have to do the following:

1. obtain the URL of the message (the post[21] described inthe previous post contains the URL - you will need to prepend https://plus.google.com to this value)
2. screen-scrape the page to obtain its message data, as described in the previous post
3. the pageData[1][15] is the XSRF token that you want

After every non-read-only operation, you will hav eto obtain a new XSRF token again. The simplest way to do it is to repeat step 2 and 3. But typically, the next XSRF token is also present in all server responses from state-modifying operations. Identifying these is left as an excercise to the reader (Firebug is your friend here).

Now that we have the required XSRF token, posting a comment to the Google+ message is very easy

jQuery.support.cors = true; 

var data = 'itemId=' + postId + '&text=' + encodeURIComponent(text);

data = data + '&clientId=os:' + postId + ':' + new Date().getTime();

data = data + '&at=' + encodeURIComponent(xsrf) + '&' 

jQuery.ajax({
        type: 'POST',
        url: 'https://plus.google.com/_/stream/comment/?_requid=' + new Date().getTime() + '&rt=j',
        data: data,
        error: function (xhr, status, error) {
                if (error.indexOf(frontGarbage) == 0) {
                    // this is a success
                } else {
                    // this is a failure. You may want to display some error message
                }
        }
    });

The important elements of this code are

• postId - this is the post ID described in the previous post. It is contained in the post[8] element of the message structure
• text - this is the text of the comment that you want to post

To Be Continued

In the next installment, I will describe how to:

• retrieve a list of circles
• add or remove a +1 to the post