A Successful Software Engineer ...

It's appraisal time in my company and as usual we (as appraisers) try to remember the tasks performed by the appraisees and evaluate them on their performance - an exercise where the appraiser compares the performance of an appraisee against the expectations set.

In this context, I attempt to find an answer to a question that's lingering in my mind for quite some time now - "who is a successful software engineer" ?

One version is that a hardworking guy is more successful than someone that gets solutions working by magic / luck. Another version is that a smart way to succeed is to google for solutions all the time 'coz there is always someone else in the world that might have run into a problem that you have. Another version is that a mix of both the qualities - hardwork + smart googling is a good combination.

Here is my version - sofware, to me, in the first place is not an original science - it's a derivative. Software engineers take the best ideas from all other sciences and real-world situations and try to model solutions. This inference mandates that a software engineer should be "observant". A software engineer should be able to think at a very basic level 'coz at the end we are going to feed the approach to a machine that has zero intelligence. The moral - "get your basics right".

As I told you, software is a derivative field - I always try to mandate a s/w engineer to be up-to-date like a doctor. Now, the reason why I draw a similarity here is 'coz a doctor should be aware of latest drugs that are effective for the diseases in town. Similarly a s/w engineer that has to keep abreast of latest technologies (ofcourse in a given specialization) in order to be an effective solution provider and I don't think it's tough as we have Google (our best friend) that ensures that all the required information is just a click away.

Last and the most important quality that I think a s/w engineer should have is "perserverance". We are dealing with machines and trying to teach a machine requires a lot of patience. When talent fails, patience prevails and this is true atleast in s/w industry.

To summarize, a s/w engineer should be "observant", "up-to-date" and "persistent" inorder to be successful.

Never miss a task ...

These days I got so busy with lot of interesting things happening in my life and at work too. I don't see any motivation in thinking about anything else other than my 6 month old daughter :) With so much action that keeps one engrossed, it's quite natural that one forgets to attend to a few important things in life - things that one used to fulfill on time earlier, though. Some of the important things are - paying credit card, utility bills on time, wishing a friend or a relative on a birthday or an anniversary, etc.

Recently I ran into a situation where I forgot to maintain sufficient funds in one of my bank accounts (an account that I use to pay my home loan). I ended up paying a penalty that I would have avoided if I had check the balance in the account. I got to know of the penalty only after the bank issued me a note seeking my attention ! Now, there are various ways to remember such things. The ways could be as simple as writing down a list of "todos" on a piece of paper every day or adding reminders on your mobile phone. I find the latter to be a good option as we spend a significant part of our life with our mobile phones.

One other interesting option to keep track of tasks and get reminded is "Google calendar". Let me tell you that Google has some really cool things but I am not sure how many people make use of them. One such thing is - "I'm feeling lucky" button on a google search page which I think is the least frequently used feature on a most frequently used web page

I started using Google calendar to keep track of my tasks and get reminded over my mobile via SMS or an email and all this comes for free :)

Let me introduce you to Google Calendar:

You just need an account with Google/Gmail to be able to access Google Calendar.

Here is what you see after you log on to Google Calendar:

• Create a new calendar or pick an existing Calendar on the left.



• Open the menu that's right next to the calendar and click on Notifications as illustrated in the screenshot.

• Set up your phone for the reminders on the page that follows.

• Provide your phone number on the following screen.
  
  
  



• Add an event

• Set up the schedule (you could set up recurring events too :))
  




• Add a reminder and get notified :)

Based on the event definition (the description that you gave and the scheduling set up for the event), Google will send you an SMS.

Now tell me, is it possible to miss a task in life ? ;)

Performance analysis ...

For the past 8 months I have been working on improving the performance of applications and most of them happen to be ASP.NET web applications.Like they say, performance analysis and tuning is an iterative process. There are no shortcuts to improve the performance of an application.

Before I reveal a couple of tools that help developers identify performance bottlenecks in their applications, let me first talk about a few basic things - most often we (software developers) get feedback saying "the application is too slow (it sucks !)". Now, this is obviously not a welcome feedback. Novice developers might quickly say -"let's comment out all the time consuming code and the application is all set ;) " Well, that's not a bad idea, rather is an option at times.
Okay, what are the areas that a developer should concentrate on when we observe that applications are performing badly - here is the list:

• Memory usage and leaks
• CPU usage
• Amount of I/O (writing to disk, database, sockets, etc)
• Time to render the application UI (applicable to windows, web applications)

I tried the following tools and this post aims at helping those developers analysing and fine tuning the performance of applications:

Name of the tool	Purpose
VMMap	VMMap is a process virtual and physical memory analysis utility. It shows a breakdown of a process's committed virtual memory types as well as the amount of physical memory (working set) assigned by the operating system to those types. VMMap is the ideal tool for developers wanting to understand and optimize their application's memory resource usage.
MemProfiler	.NET Memory Profiler is a powerful tool for finding memory leaks and optimizing the memory usage in programs written in C# or any other .NET language. It helps in identifying memory leaks by collecting and comparing snapshots of .NET memory
ANTS Memory and Performance Profiler	Available as part of .NET Developer Bundle from RedGate, this is the most powerful and user-friendly performance profiler that can profile the CPU and memory utilization of a .NET application (including locally hosted WPF XBAP applications)
HTTP Watch, Firebug, YSlow	These tools help capture the time it takes to download content from the web server to the client. These tools are applicable when profiling web applications.

Parameter driven decisions ...

It's been about 7 years since I started taking technical interviews and everytime I walk in to the interview room with a predefined set of parameters that I plan to evaluate the interviewee on. To me, an interview should be to evaluate a person on various parameters,define an acceptance criteria and thus determine if the interviewee has made it through the interview.

I learnt the art of interviewing from my seniors and so far I have been successful in picking the talent that is relevant to the kind of job we do.It sounds foolish to me to evaluate a person on hi-fi concepts when the actual work doesn't involve applying any such concepts !

In this post, I would like to talk about the homework that I do w.r.t parameters and the way I determine if a person fits in my organization/team.

There are a bunch of parameters that I enlist and the set of parameters differ based on the person that I am going to interview.For example, for a fresh grad - it might not make sense to question on software design concepts while an experienced resource is expected to know such concepts.

Since I only interview software developers/leads, here are a bunch of parameters (a 4-point formula)that I consider during my interview:

Analytical ability / problem solving skills

This is the very basic skill that we look for in a software engineer. We throw a problem that has low/medium complexity and we expect the interviewee to explain the approach to solve the problem followed by an implementation in his/her favorite programming language.

With this parameter, I can determine if the person

• understands the problem statement clearly
• is clear with what needs to be achieved
• has no assumptions
• has a clear approach to solve the problem
• is comfortable in mapping the approach to programming logic
• is comfortable writing code
• can think like a program, validates the logic, debugs and fixes the program logic to get it right

Communication skills

We expect the interviewee to understand what we are trying to see (hoping that we are good at explaining things ;)). After we are done with our explanation of a problem statement, the interviewee has only two options - he/she got the problem right or he/she has "n" number of questions to be clarified. Now, verbal communication skill plays an important role 'coz the person has to explain what the gray areas are, explain the approach that he/she plans to follow towards solving the problem.

Eye-for-detail

Most of the time, we ask people to talk about something that they worked on that made us feel excited about. Such a question serves two purposes:

1. The person tries to explain the technical challenges during the task and the way he/she went about overcome those challenges.
2. We can pick some items from his project and check to see if the person paid significant attention to the programming constructs that were used while solving a problem. Although most of us rely on Google for reusable code snippets, we expect the interviewee to have known what that "reusable code" does. (Things can't work like magic all the time, isn't it?) We expect the resources to know what they do and do (the best in) what they know

Quest for smarter solutions and thirst for knowledge

I believe in the line - "you snooze, you lose" which means that you are no longer required if you have not updated yourself on the latest developments in your relevant competancy.

We try to find out how the inteviewee tries to be up-to-date and if the guy is stuck in age old programming practices, we simply say "sorry" ;)

I feel that no matter how many rounds of interview you conduct or how many questions you prepare to evaluate a person, your parameters are going to be in the above list (more or less).

Once we fish out resources with our 4-point formula, the rest is up to HR and senior management :D

Welcome to my team :)

Steps to create stored procedure that can be used for custom pagination, sorting and filtering

In this post, let's see how to use the tool that talked about in my earlier post - Dynamic SQL for custom pagination ... to generate the sql stored procedure script.

1. Define columns in the result as follows. Check the boxes Allow Filtering and Allow Sorting if only the columns need to participate in filtering and sorting logic respectively

   

2. Click Generate Code menu to generate the stored procedure. The menu item launches a File Save dialog for you to save the generated script to a file folder.
3. Here is how the generated SQL script looks like:
   

   
   /*******************
   * Operator Enums
   *  Conditions Enum :
   * -1 - No filter
   *  0 - Null
   *  1 - Not Null
   *  2 - Equals
   *  3 - NotEquals
   *  ---- for strings ---
   *  4 - StartsWith
   *  5 - EndsWith
   *  6 - Contains
   *  ---- for int, decimal, datetime ---
   *  7 - GreaterThan
   *  8 - GreaterThanOrEquals
   *  9 - LessThan
   *  10 - LessThanOrEquals
   *******************/
   IF EXISTS (SELECT * FROM dbo.sysobjects
    WHERE id = object_id(N"[dbo].[<procedure_name>]")            AND type in (N"P", N"PC"))
    BEGIN
   DROP PROCEDURE [dbo].[<procedure_name>]
    END
    GO 
    SET ANSI_NULLS ON
    GO
    SET QUOTED_IDENTIFIER ON
    GO
    
    CREATE PROCEDURE [dbo].[<procedure_name>](
   -- TODO: Add more parameters if needed
      @whereClauseXML NVARCHAR(4000) = NULL
    , @startInd Int = 1
    , @endInd BIGINT = 5000
    , @pageSize int = 10 
    , @sortColumn nvarchar(100)  = ""
    , @sortOrder nvarchar(100) = "ASC"
    
    )
    AS
    BEGIN
    
          SET NOCOUNT ON
   
   DECLARE @idoc int
   IF @whereClauseXML IS NULL
   BEGIN
          SET @whereClauseXML = "<ROW><Filter
   FirstNameOperator="-1"
   LastNameOperator="-1"
   DateOfBirthOperator="-1"
    ></Filter></ROW>"
   END
   EXEC sp_xml_preparedocument @idoc OUTPUT, @whereClauseXML
   
   CREATE TABLE #tmpForWhereClause
   (
   FirstNameOperator INT NOT NULL,
   FirstNameValue nvarchar(50) NULL,
   LastNameOperator INT NOT NULL,
   LastNameValue nvarchar(50) NULL,
   DateOfBirthOperator INT NOT NULL,
   DateOfBirthValue datetime NULL
   )
   CREATE TABLE #FinalResults(
          [RowNum] [bigint] NOT NULL IDENTITY(1,1),
          Id uniqueidentifier  NULL ,
          FirstName nvarchar(50)  NULL ,
          LastName nvarchar(50)  NULL ,
          DateOfBirth datetime  NULL
   )
   
   INSERT INTO #tmpForWhereClause
   SELECT 
          ISNULL(FirstNameOperator,-1),
          FirstNameValue ,
          ISNULL(LastNameOperator,-1),
          LastNameValue ,
          ISNULL(DateOfBirthOperator,-1),
          DateOfBirthValue
   FROM   OPENXML (@idoc,"/ROW/Filter",1) WITH (
   FirstNameOperator INT,
   FirstNameValue nvarchar(50)
   ,LastNameOperator INT,
   LastNameValue nvarchar(50)
   ,DateOfBirthOperator INT,
   DateOfBirthValue datetime
    )
   
   --------------------- BEGIN -----------------------------
   ----   Create a temp variable or a CTE with the actual SQL search query ----------
   ----   and use that CTE in the place of <table> in the following SQL statements ---
   ----          or  ---
   ----   If you want to wrap this stored proc around an existing stored procedure then,
   --    Create a #temp table ---
   --    Execute the following SQL –
   --    INSERT INTO #tmpResults
   --    EXEC <your_existing_sp> @param1, @param2 ….
   --------------------- END -----------------------------
   INSERT INTO #FinalResults
   SELECT
          T.Id,
          T.FirstName,
          T.LastName,
          T.DateOfBirth
   FROM <table> T,
   #tmpForWhereClause TMP
   WHERE (
   
   (
           ( TMP.FirstNameOperator = -1 )
    OR
           ( TMP.FirstNameOperator = 0 AND T.FirstName IS NULL )
    OR
           ( TMP.FirstNameOperator = 1 AND T.FirstName IS NOT NULL )
    OR
           ( TMP.FirstNameOperator = 2 AND T.FirstName = TMP.FirstNameValue )
    OR
           ( TMP.FirstNameOperator = 3 AND T.FirstName <> TMP.FirstNameValue )
    OR
           ( TMP.FirstNameOperator = 4 AND T.FirstName LIKE TMP.FirstNameValue + "%")
    OR
           ( TMP.FirstNameOperator = 5 AND T.FirstName LIKE "%" + TMP.FirstNameValue )
    OR
           ( TMP.FirstNameOperator = 6 AND T.FirstName LIKE "%" + TMP.FirstNameValue + "%" )
    )
   
   AND
   
   (
           ( TMP.LastNameOperator = -1 )
    OR
           ( TMP.LastNameOperator = 0 AND T.LastName IS NULL )
    OR
           ( TMP.LastNameOperator = 1 AND T.LastName IS NOT NULL )
    OR
           ( TMP.LastNameOperator = 2 AND T.LastName = TMP.LastNameValue )
    OR
           ( TMP.LastNameOperator = 3 AND T.LastName <> TMP.LastNameValue )
    OR
           ( TMP.LastNameOperator = 4 AND T.LastName LIKE TMP.LastNameValue + "%")
    OR
           ( TMP.LastNameOperator = 5 AND T.LastName LIKE "%" + TMP.LastNameValue )
    OR
           ( TMP.LastNameOperator = 6 AND T.LastName LIKE "%" + TMP.LastNameValue + "%" )
    )
   
   AND
   
   (
           ( TMP.DateOfBirthOperator = -1 )
    OR
           ( TMP.DateOfBirthOperator = 0 AND T.DateOfBirth IS NULL )
    OR
           ( TMP.DateOfBirthOperator = 1 AND T.DateOfBirth IS NOT NULL )
    OR
           ( TMP.DateOfBirthOperator = 2 AND T.DateOfBirth = TMP.DateOfBirthValue )
    OR
           ( TMP.DateOfBirthOperator = 3 AND T.DateOfBirth <> TMP.DateOfBirthValue )
    OR
           ( TMP.DateOfBirthOperator = 7 AND T.DateOfBirth > TMP.DateOfBirthValue )
    OR
           ( TMP.DateOfBirthOperator = 8 AND T.DateOfBirth >= TMP.DateOfBirthValue )
    OR
           ( TMP.DateOfBirthOperator = 9 AND T.DateOfBirth < TMP.DateOfBirthValue )
    OR
           ( TMP.DateOfBirthOperator = 10 AND T.DateOfBirth <= TMP.DateOfBirthValue )
   
   )
   
   AND
    1 = 1
    )
    ORDER BY
           CASE WHEN @sortColumn = "FirstName" AND @sortOrder = "ASC"
          THEN T.FirstName END ASC,
           CASE WHEN @sortColumn = "FirstName" AND @sortOrder = "DESC"
          THEN T.FirstName END DESC ,
   
          CASE WHEN @sortColumn = "LastName" AND @sortOrder = "ASC"
          THEN T.LastName END ASC,
           CASE WHEN @sortColumn = "LastName" AND @sortOrder = "DESC"
          THEN T.LastName END DESC ,
   
          CASE WHEN @sortColumn = "DateOfBirth" AND @sortOrder = "ASC"
          THEN T.DateOfBirth END ASC,
           CASE WHEN @sortColumn = "DateOfBirth" AND @sortOrder = "DESC"
          THEN T.DateOfBirth END DESC
   
   
   DECLARE @count INT  
   SET @count = 0  
   SELECT @count = MAX(RowNum) FROM #FinalResults
   IF @startInd  > @count  
   BEGIN  
          DECLARE @numOfPages INT   
          SET @numOfPages = @count / @pageSize  
          IF @count % @pageSize > 1  
          BEGIN  
                 SET @numOfPages = @numOfPages + 1  
          END  
          SET @startInd = ((@numOfPages - 1) * @pageSize) + 1  
          SET @endInd = @numOfPages * @pageSize  
   END
   
   SELECT @count AS TotalRows, * FROM #FinalResults WHERE RowNum BETWEEN @startInd AND @endInd
   
   DROP TABLE #tmpForWhereClause
   DROP TABLE #FinalResults
   END
   

   
   
   
4. Update the text that reads - <procedure> or <table> in the above SQL script as you see fit.
5. Execute the stored procedure on the database and test once to ensure that it works :)

String.Find(..., pattern) in SQL server 2005 or above using CLR functions ...

SQL Server 2005 and above let developers take advantage of .NET through CLR assemblies and functions. In this post, we shall attempt to write a custom CLR function that allows us to search for a string pattern (a regular expression with named groups).

Example:

Input string - Kiran123Banda456Kiran789
If the requirement is to capture all those numbers that immediately occur after the word "Kiran", then the expected result is 123,789 (assuming that multiple occurances are concatenated with a comma).

The code for the custom CLR function is as follows:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SqlServer.Server;
using System.Data.SqlTypes;
using System.Text.RegularExpressions;


[Serializable]
public class StringSearch
{
    /// 

    /// Finds the specified input string.
    /// 
    /// The input string.
    /// The search pattern.
    /// 
    [SqlFunction()]
    public static SqlString Find(SqlString inputString, SqlString searchPattern)
    {

        StringBuilder sb = new StringBuilder();

        Match match = Regex.Match(inputString.Value, searchPattern.Value, RegexOptions.Compiled);
        bool matchResult = match.Success;

        do
        {
            if (match.Groups.Count > 0 && match.Groups["grpVar"] != null)
            {
                for (int i = 0, l = match.Groups["grpVar"].Captures.Count; i < l; i++)
                {
                    string trailingDigit = match.Groups["grpVar"].Captures[i].Value;
                    sb.Append(trailingDigit);
                    sb.Append(";");
                }

            }

        } while ((match = match.NextMatch()) != null && match.Success);

        return new SqlString(sb.ToString());

    }
}

Steps to deploy and use the custom CLR function

Copy the attached dll to a location, say, E:\tmp and run the following sql statements to register the custom assembly and the user-defined function

CREATE ASSEMBLY CustomAssembly FROM 'E:\tmp\CustomAssembly.dll'
GO

CREATE FUNCTION fnSearchString(@input nvarchar(1000),@searchPattern nvarchar(100)) RETURNS nvarchar(max) AS
EXTERNAL NAME CustomAssembly.StringSearch.Find
GO

exec sp_configure 'clr enabled','1'
Reconfigure with override

Example Usage :

DECLARE @inputString NVARCHAR(MAX)
DECLARE @searchPattern NVARCHAR(255) 
SET @inputString = 'Kiran123Banda456Kiran789'


SET @searchPattern = 'Kiran(?\d+)'

SELECT [dbo].[fnSearchString](@inputString,@searchPattern)

Output :

123,789

Dynamic SQL for custom pagination, filtering and sorting ... (without using string concatenation !) ...

Over the past few weeks, I spent time in optimizing existing applications and improving the performance. "Performance" is a big word and one can realize how long a "second" is while tuning an application for performance. I had optimized .NET code as well as SQL queries.

Developing web applications for performance and writing apps/solutions that can survive security require a lot of thinking and experience.

One of the interesting requirements that I spent time on very recently is related to SQL and the requirement is as follows:

Consider an application that presents the data in a grid (a table) where a user can sort the data, apply filters and page through the data pages. Here is a screenshot that better explains the concept visually:

Assuming that the stored procedure returns 1000 rows and only 10 records are shown per view in the grid. This means that there are about 100 pages and 10 records per page.

Now, there are a number of ways to bind data to a grid and here a few options:

Option # 1:
Get all 1000 rows from database server to the application server.

Perform sorting, filtering on the in-memory resultset and pick the 10 records that relevant to the current page in the grid.

The downside of this approach is that it caches 1000 records per request although the view requires just 10 records !

Option # 2:
Apply filtering, sorting at the database and return just 10 records that are relevant to the current view in the grid.

The advantages of this option are - less data transfer between db and app server & low memory footprint on the app server (as it holds just 10 records as opposed to 1000 records)

From the above justification, it is obvious that option # 2 is efficient w.r.t performance.

With this in mind, you start googling for a solution and you will find a solution that operate as follows:

Concatenate : SELECT Query + @SortColumn + @SortOrder + WHERE RecordID BETWEEN + @StartIndex + AND + @EndIndex
where @SortOrder, @SortColumn, @StartIndex and @EndIndex are variables/parameters to your stored proc.

The above approach works and is simple to code too but it is prone to "SQL Injection attack" !!

So, the problem I tried to solve is to avoid SQL String concatenation to build an SQL query and yet achieve the required dynamism :)

I am not going to explain the logic behind the SQL query as the script is self-explanatory. I have also developed a simple windows application in .NET that lets you generate most of the SQL query with placeholders that can be changed to suit your requirement. All that you have to do is define the columns for which the filtering and sorting logic is going to be dynamic.

Try the tool once and get back to me if you run into issues.