Testing Thoughts

I am not the original author of this post’s contents.

I recently found this email reply from John Mitchell while discussing the testing aspects of micro services in our project. It is so informative that I didn't want this to get lost in emails. Adding this here.

(Names have been changed to keep the focus on the topic under discussion)

Hi PersonA,

I think our discussions about the Quality Criteria were pretty clear…

• And to be even more precise, they must be able to be automatically run as part of the appropriate phases in the pipeline.
• APIs
• Message / file formats, etc.
• UI’s which they provide
• Telemetry which they generate for business/application/operation/etc. monitoring/alerting/tracking/etc.

Martin’s slide that you pointed is no excuse to get out of any of the above. He’s pointing out, as I did, that false metrics such as “code coverage” percentages are misleading and inefficient ways to measure actual quality of the software (as opposed to trying to force the team to spend time creating “tests”).

Again, as we discussed in our last meeting, if you achieve anything even remotely close to my list above (and on the quality criteria page) you will have created software that is quite high quality.

As I noted, people must stop over-indexing on this so-called end-to-end testing to the exclusion of the rest of the criteria. That’s what’s lead to the current mess that we have with the AppA bundle today. It’s precisely the fact that each of those various angles/levels of testing are done (automatedly) that provide what we have talked about in terms of being able to bracket problems for much faster triage and debugging, achieve the sorts of testability for multiple uses, provide reality based assurances to InfoSec, the app teams, the management, etc. that are stuff really does work.

Thanks,

John

Read More

Using GCViewer For GC Analysis

GCViewer developed by Tagtraum Industries is one of the free tools that you can use to visually analyse the garbage collection logs.

Running GCViewer

The latest version is now maintained by chewiebug and available for download at http://sourceforge.net/projects/gcviewer/files/

• Download the gcviewer-x.xx.jar to your local system
• Launch the viewer by running java -jar path_to_gcviewer-x.xx.jar
• You need Java version >= 1.7

Using GCViewer

• Open the GC log file from the test run using GCViewer
• Adjust the zoom levels (dropdown at the top) so that the graph fits the window and there are no scroll bars (to get an overview)

• Check the trend of the 'Total Heap' usage of the VM. As long as it does not show an upward trend, VM is considered to be fine.
• Check the right side of the tool for the information related to the run - Summary, Memory and Pause.

Summary

• 'Number of full gc pauses' are of concern and healthy VM should not generally be doing a full GC (which means it should ideally be zero). Full GCs result in the VM being inactive till the GC completes.

Memory

• The 'Total heap' gives an indication on how much the VM memory is loaded.

Pause

• Total time spend on 'gc pauses' is of interest.

Read More

macOS - Useful Utilities

Day-0

For getting the calendar along with time.
Download

BetterSnapTool

For those Windows users who is missing the maximize and automatic snapping of windows. It's worth it's price (Rs. 110).
Download

iTerm

A much better terminal than the built-in one that comes with Mac
Download

Clipmenu

Multiple clipboards to save you from losing your work.
Download

Skitch

Much better screen snapping tool.
Download

Read More

Scrum - Collection

 

Read More

Time Estimates and Story Points

I would like to share an excellent write-up by the JIRA Agile Product Manager from Atlassian Shawn Clowes. This was written a while ago, but the depth and detail present in this reply from Shawn for one of the questions in the Answers site still amazes us. This was his explanation about time estimates and story points in the agile context.

Here it goes:

I'd like to provide a full explanation of why we we've offered 'Original Time Estimate' as an 'Estimate' value and not 'Remaining Estimate'. Some of my discussion refers to agile concepts that anyone reading probably knows well but I've included it because the context is important. Note that the discussion refers to the best practices we've implemented as the main path in GreenHopper, you can choose not to use this approach if you feel it's really not suitable.

Estimation is separate from Tracking

In Scrum there is a distinction between estimation and tracking. Estimation is typically performed against Primary Backlog Items (PBIs, usually stories) and is used to work out how long portions of the backlog might take to be delivered. Tracking refers to monitoring the progress of the sprint to be sure it will deliver all of the stories that were included. Tracking is often performed by breaking down stories in to tasks and applying hour estimates to them during the planning meeting then monitoring the remaining time in a burndown during the sprint. 

Estimation is all about Velocity

The primary purpose of applying estimates to the PBIs is to use that information to work out how long it will take to deliver portions of the backlog.

In traditional development environments teams would estimate items in 'man hours' and these would be assumed to be accurate. They could then count up the hours in the backlog for a project, divide by the number of people on the team and hours in the week to reach a forecast date. Of course, these estimates often proved to be wildly inaccurate because they did not take in to account the natural estimation characteristics of the team (for over/under estimation), unexpected interruptions or the development of team performance over time. The inaccuracy of the estimates combined with the significant cost of the time spent trying to 'force' them to be accurate makes the 'man hours' approach difficult if not impossible to make work.

So in the Scrum world most teams do not try to achieve estimation accuracy, instead they aim to achieve a reliable velocity. The velocity is a measure of the number of estimation units that a team tends to complete from sprint to sprint. After their first few sprints most teams will achieve a reasonably consistent velocity. Armed with velocity and estimates on the PBIs in the backlog teams can look forward to predict how long portions of the backlog will take to complete.

The key is that it does not matter what the estimation unit is, just that from sprint to sprint it becomes reasonably predictable. For example, teams can choose to use 'ideal hour' estimates but it's neither necessary or expected that those hours will have any close relationship to elapsed time. If a team has 'man hour' capacity of 120h in each sprint but a velocity of 60h that makes no difference because the 60h velocity can still be used to estimate the number of sprints that portions of the backlog will take to complete and therefore the elapsed time. Many people then start wondering where 'the other 60 hours' went and implying that there is something wrong with team productivity. But that's usually got nothing to do with it, a team's estimates merely represent their view of how hard items will be and they're always polluted by the team's natural behaviour (for example over/under estimation) as well as organisational overhead etc. The velocity is all that matters from a planning perspective.

Since the units are not related to time, most teams now choose to use story points (an arbitrary number that measures the complexity of one story relative to others) as their estimation unit. Story points clearly break the mental link with time.

Inaccurate Estimates are good, as long as they are equally Inaccurate

Velocity will only reach a stable state as long as the team estimates each backlog item with the same level of accuracy. In fact, it's probably better to say that each item should be estimated to exactly the same level of inaccuracy. At the risk of repeating the obvious, the goal of velocity is to be able to look at a backlog of not particularly well understood stories and understand how many sprints it will take to complete. This requires a similar level of uncertainty for all of the estimates that are in the backlog.

One of the counter intuitive implications is that teams should estimate each item once and not change that estimate even if they discover new information about the item that makes them feel their original estimate was wrong. If the team were to go ahead and update estimates this 'discovery of new information' will happen regularly and result in a backlog that has some items that have higher accuracy but most that don't. This would pollute velocity because sprints with a larger percentage of high accuracy estimates will complete a different number of units compared to those with a lower percentage of high accuracy estimates. As a result the velocity could not be used for its primary purpose, estimating the number of sprints it will take for a set of not well understood stories in the backlog to be completed. Therefore it's critical to use the first estimates so that the team's velocity realistically represents their ability to complete a certain number of units of not well understood work far ahead in to the future.

But what about when teams realise they've gotten it wrong?

Consider the following scenario:

• Issue X has an original estimate of 5 days.
• The issue's estimation was too optimistic and they realise it's actually 15 days before the next sprint is planned.

Some people would argue that using the original estimate will endanger the sprint's success because the team will take in what they think is 5 days of work in to the next sprint when it's actually 15

However, the inaccurate estimate of 5 days is unlikely to be an isolated occurrence, in fact the estimates are always going to be wrong (some very little, some wildly so). Often this will be discovered after the sprint has started rather than before. As long as the team estimates the same way across the whole backlog this will work it self out over time. For example if they always underestimate they may find that for a 10 day sprint with 4 team members they can only really commit to 20 days of their estimation unit. If they have established a stable velocity this has no effect because from a planning perspective we can still estimate how much work we'll get done in upcoming Sprints with good certainty.

But doesn't that break the Sprint Commitment?

When it comes time to start a sprint, the team can use the velocity as an indication of items from the backlog they can realistically commit to completing based on the amount they have successfully completed in the past. However, many people immediately question how that can be right when the original estimates will not include information about work that might have already been done or discovered information about how hard the work is.

As an example, consider the following scenario:

• An issue has an original estimate of 10 days.
• The team works 5 days on the issue in the current sprint.
• The team discovers a bad bug somewhere else in the project and decide that fixing that bug in the current sprint is far more important than completing issue X as planned.
• The sprint gets finished and the issue returns to the backlog.

In the next sprint the team would be tempted to update the estimate for the issue to 5 days and use that to make their decision about whether to include it in the sprint. The implication is that they might not include enough work in the next sprint if they used its original estimate of 10d. However, the reason that the task was not completed previously is because of unplanned work, it's unrealistic to assume that won't happen again in the future, perhaps even in the next sprint, thus the 10d is a realistic number to use in absence of certainty. As a result the cost of the unplanned work that may happen is eventually accounted for in the original estimate. Even if the work does turn out to be insufficient for the next sprint the team will correct that by dragging more work in to the sprint.

In the same example, consider if this were the only issue in that sprint and will be the only issue in the next. If the issue is completed in the second sprint and we use the remaining estimate the velocity will be (0d + 5d) / 2 = 2.5d, but the team can clearly complete more work than that in future sprints. If we use the original estimates the velocity will be (0d + 10d) / 2 = 5d. The use of the original estimate accounts for the fact that the team cannot commit to 10d in every sprint because unplanned work will likely make that impossible, it also realistically accounts for the fact that unplanned work will not happen in every sprint.

Why not estimate on sub-tasks and roll that up for Velocity and Commitment?

Many teams break down stories in to sub tasks shortly before the sprint begins so they can use the stories for tracking. This raises the possibility of using the sum of the estimates on the sub-tasks as a way to decide which issues to commit to in the sprint (and potentially for velocity).

As described above, tracking is really a separate process from estimation and velocity. The estimates that are applied to the sub tasks are clearly higher accuracy than those that were originally applied to the story. Using them for velocity would cause the velocity to have both high and low accuracy estimates, making it unusable for looking looking further out in the backlog where stories have only low accuracy estimates. In addition, only items near the top of the top of the backlog are likely to have been broken to tasks, so using task estimates for velocity means that the velocity value could only ever predict the time to complete the backlog up to the last story that has been broken in to tasks.

Using the sub task rollup to decide the sprint commitment would also be dangerous because unlike the velocity value it does not take in to account the overhead of unplanned work or interruptions.

Conclusion

Many industry leaders are moving away from hour estimates of any sort. This makes sense because the main questions to be answered are 'How much work can we realistically commit to completing this sprint?' and 'How long will this part of the backlog take to deliver?'. A story point approach based on original estimates can deliver the answers to these questions without the anxiety around 'accuracy' that teams feel when asked to estimate in hours.

The GreenHopper team itself uses the approach described in this article and has established a reliable velocity that we have used to plan months in advance, even when new work has been encountered during those months.

We recommend this approach because while it is sometimes counter-intuitive it is also powerful, fast and simple.

All of that said one of the key precepts of Agile is finding the way that works for you. So GreenHopper does support the alternatives described above including the use of remaining estimates for sprint commitment, hours for estimation and hour estimates on sub-tasks.

Read More

How To Become A Famous Blogger

 

Read More

How Popchrom And Lazarus Saves My Day

What is Popchrom and Where To Get It

Save time and effort with Popchrom by creating your own shortcuts for text phrases! Whether it's a simple email signature or…

To transform your abbreviation place the cursor on it and hit your personal shortcut (default is 'ctrl + space').

Get Popchrom

What is Lazarus and Where To Get It

Lazarus Form Recovery is a free browser add-on for Firefox, Chrome and Safari that autosaves everything you type into any given web-form. We've all had the frustrating experience of spending ages getting a form entry just right, only to suffer rage and disgust when all that hard work is destroyed, whether it's a website timeout, a browser crash, a network failure, or just a cat wandering across the keyboard. Lazarus makes recovering from such mishaps painless in most cases, and we're working on the rest of them!

UPDATE:

Lazarus is withdrawn and now we have Typio

Read More