Manthan

I have bitten by “App Out of Quota” error on google app-engine more than once. So it become critical for me to load test all my apps before publishing them to the rest of the world.

Looking around , Loadstorm offers a simple solution to load testing. It a web-service with a free and premium service. Up-to 50 concurrent users is free and there are different tiered pricing if you want more.

Load Storm, has a simple UI where you can

• Authenticate your website
• Create test steps
• Schedule one or more load testing session etc.

Microsoft has a new announcement in the area of "hosted computing” called “azure”. Its a new service rivaling amazon’s webservices and Google’s half baked Appengine.

The problem with this announcement is, the signup page doesn’t take me anywhere nor can I find any “sample” sites that are running azure. It looks like a work in progress and will be sometime before it is released to the developers. I just wish they(Microsoft) make their websites simple and not generate so much of noise, with no significant information. Make an announcement if it is ready and I can take it for a spin.

The previously released Microsoft SSDS ( Sql server data services) is a part of this whole azure tech stack.

• Windows Azure for service hosting and management, low-level scalable storage, computation and networking
• Microsoft SQL Services for a wide range of database services and reporting
• Microsoft .NET Services which are service-based implementations of familiar .NET Framework concepts such as workflow and access control
• Live Services for a consistent way for users to store, share and synchronize documents, photos, files and information across their PCs, phones, PC applications and Web sites
• Microsoft SharePoint Services and Microsoft Dynamics CRM Services for business content, collaboration and rapid solution development in the cloud

One of the most significant advantages of Azure is going to be support for “ALL” languages. I don’t know when this is going to happen, but is something they are promising. Support for PHP, Rails etc is fundamental for the success of azure , because of the body of open source code already existing for these tech stacks. It will be quite simple to port the existing PHP application on-to azure when it is available.

One of the biggest problem with Appengine is, there are no “open source” code base which can be leveraged to build apps. Simple functionality, which is normally pulled off the web and implemented with a copy paste, has to be laboriously hand-coded for Appengine.

Just hope azure serves as wake up call for Appengine team and they speed up their releases.

I built a couple of applications using Google Appengine. The google appengine was marketed as highly scalable with high-availability. Google Appengine looked like an ideal choice for “salary search” application that I was planning to build for a while. The data-set that I was looking at was more than 2 Million records to begin with and felt that Appengine could scale well. Two weeks after releasing it, I am frustrated with how little Google Appengine delivers on its promises.

1. Abrupt time-out errors: This is the single biggest issue with Google Appengine. Nobody including Google has explained what causes this and how to avoid these abrupt “Google Appengine is Over Quota” errors. We are not talking “Digg” effect, but something like 3K page views and 100 users per hour. Google appengine couldn’t scale to handle that kind of load.
   • Not doing any writes to the datastore which are more expensive.
   • Every request is memcached
   • Not more than 100 entities are being retrieved per request.
2. 1000 Entity retrieval limit: There is a limit on how many entities of a particular type can be retrieved. For example, when I try to list all the jobs offered by Infosys, the results cannot go beyond a count of 1000. There is a offset parameter but that offset works within this 1000 entities limit. So if the offset exceeds 1000, the query bombs.
3. SDK is broken: The SDk and the production versions are out of synch. What works in the SDK does not necessarily works in PROD version.
4. Indexes: Building Indexes takes more than 24 hours if it is successful.
5. Updating Data is cumbersome: Once the data is uploaded, its very difficult to update that data. There are no “bulk” update operations that can be performed on the data. The only way I am aware of is creating a page and set it to “auto-refresh” using Meta-tag. The data operations functions are so rudimentary that no serious iterative development can be done on app-engine. Small changes to the data-model can take fore ever to implement.
6. Documentation and Support: Documentation is limited. For support one has to be active on google appengine group. Though the engineers on the group try their best, the support needs are lot of improvement
7. CRON/Batch jobs not supported: There is no support for CRON jobs or batch processing.

In conclusion, Appengine is Work in Progress and it has a long way to go before it can live up to its expectations. It is definitely not suited for iterative development, where companies want to release incremental functionality regularly. If the data-model is not well thought out , then Appengine framework will lock all your data or not scale at all, which defeats the purpose.

Google Appengine does not provide a straight forward way to upload data to the data-store. They have one article which indicates a way to upload data to Google Appengine data store. Using this method, one can upload CSV file to your google Appengine APP by making repeated “HTTP” requests. While this work for smaller data-sets (1000) records or less, it fails for bigger data sets. I had a requirement to upload mover 2 million records for my salary search app and the program kept failing. Resuming a bulk upload, resets the counter and it start all the way from the beginning causing duplicate data.

I have modified the bulk upload program to provide do the following things

1. Introduced a parameter called “skip” which will skip “N” number of records when the bulk data uploader resumes
2. A lot of times, the bulk data uploader fails because of either time-out issues or network issues. Instead of aborting in such scenario, the program just goes to sleep for 1 min before resuming from where it left.
3. This program writes the progress log to a text file. So in case of a failure, the log file contains the entry to where the program failed last time. You can skip that many records when resuming.
4. Sample Call :

bulkload_client_resume.py –filename upload_me.csv –kind Visa  –url .appspot.com/load">.appspot.com/load">http://<your app>.appspot.com/load  –skip 1000

In order to resume the Bulk Data uploader, you have modify the following sources in your SDK

• bulkload_client.py under Google\google_appengine\tools
  • Make a copy of the existing file and create a new file with the name bulkload_client_resume.py
  • Change following line of code in bulkload_client_resume.py

          BULKLOAD_CLIENT_PATH = ‘google/appengine/tools/bulkload_client_resume.py’

• bulkload_client.py under the following folder Google\google_appengine\google\appengine\tools
  • Make a copy of existing file and create a new file with the name bulkload_client_resume.py
  • Modify ImportCSV function
  • Introduce a new parameter called skip, which will skip "X” number of records before resuming upload.

def ImportCSV(filename,
              post_url,
              cookie,
              batch_size,
              kind,
              skip,
              split_url=SplitURL,
              openfile=file,
              create_content_generator=ContentGenerator,
              post_entities=PostEntities):
  """Imports CSV data using a series of HTTP posts.

  Args:
    filename: File on disk containing CSV data.
    post_url: URL to post the Entity data to.
    cookie: Full cookie header to use while connecting.
    batch_size: Maximum number of Entity objects to post with each request.
    kind: Entity kind of the objects being posted.
    split_url, openfile, create_content_generator, post_entities: Used for
      dependency injection.

  Returns:
    True if all entities were imported successfully; False otherwise.
  """
  host_port, uri = split_url(post_url)
  csv_file = openfile(filename, ‘r’)
  retry=1
  i=0
  try:
    content_gen = create_content_generator(csv_file, batch_size)
    logging.info(’Starting import; maximum %d entities per post’, batch_size)
    for num_entities, content in content_gen:
      retry=1

#logic to skip
      if i<int(skip):      
       logging.info(’Skipping  entities in %d skip %d’, i, int(skip))      
       i=i+int(batch_size)
      else:
       logging.info(’Importing %d entities in %d bytes current count %d’,
                   num_entities, len(content),i)
       LogMessage(’Importing  entities in  bytes current count ‘+str(i) +’\n’)
       LogMessage(time.ctime(time.time()) +’ :Importing  entities in  bytes current count ‘+str(i) +’\n’)      
       while retry==1:
               try:
                 content = post_entities(host_port, uri, cookie, kind, content)
                 retry=0
                 i=i+int(batch_size)
               except PostError, e:
                 logging.error(’An error occurred while importing: %s’, e)
                 logging.error(’Going to sleep’)
                 LogMessage(time.ctime(time.time()) + ‘ ERROR Count:’+str(i))

                #Sleeps for 1 min before retrying the same record       
                 time.sleep(60)
                 logging.error(’Retrying %s’)
                 retry=0
  finally:
    csv_file.close()
  return True

• Introduce a new function, which will write a upload log to a log file.

def LogMessage(message):
try:
  logfile = open(‘bulkupload.log’, ‘a’)
  logfile.write(message)
except:
  logging.error(sys.exc_info()[0]);
  if logfile:
   logfile.close()

Feel free to drop a note if you have trouble getting this to work

Have created a new “salary search” application using Google’s Appengine. The salary search is based on data released by department of labor. The size of the data is large and I have upto 2 million job listings to search from. Considering the size of the data, Google Appengine sounded like a good Idea.

Some interesting searches from the search app listed below

Tech Companies:

1. Google Engineer Salary
2. Microsoft Engineer Salary
3. Apple Exploratory Design Engineer

Consulting Companies:

1. Bearing Point Salary
2. KPMG
3. Ernst & Young

Having worked in different companies on both side of the ocean, I had been mulling over what makes a company succeed in outsourcing. What to outsource and factors to consider before outsourcing. Listed below is a summary of what works and what doesn’t

Commitment, motivation and security: Commitment to outsourcing from the whole team is absolutely required for success of any project. Its vital that the current team looks at Outsourcing as away to augment their strength. Its very easy for organizations to get into US vs Them situation, creating un-necessary conflicts. These can be limited by making sure that the current team is secure in the belief that that outsourcing is a way to compliment their work and not to replace them. Travel, more face time, flat organizations help a lot in these areas.

Competency loss in US: There should not be any loss of competency as a result of outsourcing. The whole idea is to increase competency and allowing the organization move further, faster. The team in US/HQ should be so strong, that if tomorrow the IT location in Asia has to be closed, the team in US can pick up the baton and run with it.

What to outsource ? : Non core business areas. IT , help-desk, admin work, testing, development etc. Any area of work , which doesn’t add value to the business the company in.  Start with those areas and evaluate. Start with support work, then with projects that are autonomous. Say development of tools like time-card or employee expenses etc. There is always a shortage of DBA’s and these can be outsourced to get 24/7 coverage. Helps the DBA’s in US and they appreciate to off the hook at nights/vacations. Target job reqs that are open for years and an equivalent resource could not be found in the US.

Cost: Evaluate the cost every quarter and analyze the cost advantage that the company gains by outsourcing. Is it worth it ? There are some in-tangible costs to outsourcing, like more layers of management. Extra management time spent by leadership, to make sure outsourcing is successful. The salaries in India raise at a much higher rate , compared to US. Cost of outsourcing is a moving target and has to be re-evaluated every quarter. With inflation at 11.5% in a country like India, there is only this much that anybody can do, to ensure that cost remain in control.

Attrition: Be prepared for higher attrition and the possibility that the some of the resource end up coming into US and start working for a competitor. Working at the same “Title” for five years is unheard of in India. Resources in India are very highly mobile, very motivated and are always seeking new challenges. Therefore there is a high possibility that they would leave the company in span of 2 years and work for a competitor in the US.

Innovation cannot be outsourced: If your core business is innovation, outsourcing would be suicidal. You cannot outsource innovation, not even to the next state, let alone a distant country. What ever success the company has today, is a direct result of leadership, innovation and hard-work of the current / past employees and that DNA cannot be transferred to another city/country so easily. The reason innovation thrives in a place like Silicon Valley is because of the eco-system there fosters innovation. That is a reason, people from India thrive in Bay Area and start great business, great product companies.

• Have you heard of any product company from Bangalore?
• How many patents were filed from India in FY08 ?
• There is no question about the quality of resource from Asia. They might be even better, but does the environment in that country or that regional office foster the kind of innovative environment that the company needs?
• Is it easy to replicate the same model there ?
• These questions have to be answered with an Emphatic YES before the decision on outsourcing is taken.

Loss of Resource Advantage: It is very tempting for any company to reduce costs by freezing hiring in the HQ and hiring in Asia. But every potential great candidate you fail to hire in the US is a potential competitor. Every MVP who doesn’t get drafted is a potential game winner for the competitor few years down the line.The current level of success of any company is in no small measure to these heavy hitters, who changed the game. Companies are always competing for the best from the next generation. One should guard against the inclination to hire in Asia at the cost of potential MVP’s in the US. In the battle of resources, companies like Google hire the best, even if they have no defined work for them. Its great to have self-motivated people, because they will find ways to prove their worth and value.

Evolution: Outsourcing is a more like a 10 year plan. It has to be gradual, organization has to experiment and learn at each and every move. Case in point, Oracle India. Started in 1994 , but it really grew in the year 2001, when the economy was down and outsourcing got the management blessings. Today it can be considered the gold standard of outsourcing, with 10K plus employees in the India in almost every domain area. Oracle started with outsourcing Support and Staff augmentation for their consulting business. Over period of time, they improved the process , learnt the best practices and moved to development. Today lot of the teams are self-sufficient and are able to develop quality products based on designs developed in HQ.

Strong Leadership: This is the single most important aspect for success of outsourcing. There has to be strong leadership on both sides of the ocean and preferably old timers. People who have the company DNA in them and can replicate it in the outsourced location. Going out and hiring managers from the market is going to dilute the work culture, adaptation is going to difficult . Depute leaders to India, put them in charge of hiring, mentoring and building teams. Have close interactions, by encouraging travel in the beginning. Unless these are done right, there is a risk that you end up running two different companies , one in Us and one in India. Create an environment to encourage leadership and pick managers in the outsourced location from within the ranks. Once this starts happening, then the organization is almost on Autopilot.

Strong Process: Does the system you are trying to outsource have a well defined process ? Is it too people dependant ? If Yes, then its not a candidate for outsourcing at this time. Outsource systems which are people independent. Management should be tweaking the system and not spending time managing people. Put self-managed people to run the system in the outsourced location, who are go-getters and have proven experience in doing this successfully before.

Learn from success stories: Oracle , SAP, Microsoft, Texas Instruments, CISCO etc  are good success stories in the domain of outsourcing. Their models are worth evaluating and replicating. CISCO is especially interesting because they matured to the extent where they are doing a lot of R&D in outsourced locations. While not recommended for all companies, its a model worth looking at.

Recommended Readings

Crawl, Walk, Run Strategy Leads to Success in Offshore Outsourcing

http://whitepapers.silicon.com/0,39024759,60039273p,00.htm

Orange County Register is carrying out a pilot program of outsourcing copy editing work to India. This one month long pilot program would be on trial basis and the future direction would be determined based on the outcome of this program.

The newspaper business has been struggling to reinvent itself to compete. Now the Orange County Register is trying a new way to cut costs: outsourcing to India.

Mindworks Global Media will copy edit some of the papers stories for a one-month trial starting next week. And a community newspaper owned by the O.C. Register’s parent company–it didn’t name which one–will outsource page layout to Mindworks, which is based outside New Delhi.

The outcome of this pilot program would be keenly watched by other print publications. With the declining Ad Revenues, the print industry has to re-invent it self and look for cost reduction every where possible.

The newly unveiled iPhone 3G is going to be simultaneously launched in multiple countries on July 11th. India is amongst the list of countries which are still “wait-listed”

As per the last update from Apple,  Bharati is the preferred carrier. Unlike in the US, long tedious contracts are the not the normal and people are used to using unlocked phones. So in all probability, customers will have to fork out the entire cost of the device, probably with a marginal subsidy from Airtel for every new customers.

Google just launched a new feature on Google trends called Google Trends for website. Its a very exciting new feature for ranking websites in direct competition to Alexa etc.

But I noticed, google trend for google doesn’t feature the traffic data for google properties like google.com, orkut.com

So comparisons like google.com vs yahoo.com will not work.

With the excitement surrounding Firefox 3 launch, I gave in to the temptation and upgraded to Firefox 3.

To my shock, I realized how much I depended on firefox and how it has become a virtual OS for my daily needs when none of the plugins worked. I use the following plugins daily

1. Grease Monkey
2. Google Tool Bar
3. S3 Organizer for Amazon S3
4. FireBug
5. FireFTP

These plugins are absolutely essential for my work. I have got so used to not installing software , instead installing plugins for Firefox, that Firefox is the new OS and upgrading firefox is like upgrading the OS.

So make sure your plugins are supported before upgrading Firefox.

Now about the downgrading process. Un-install Firefox 3, then download the previous version of Firefox from Mozilla releases website.

When I just installed the release 2 over the existing Firefox 3 release, the browser just wouldn’t open up. It kept on crashing. So make sure you un-install the existing Firefox 3 installation and then install firefox 2. The only saving grace is, all the plugins are intact, inspite of the un-installation process.

Update : One of the readers  pointed out that FireBug works in Firefox 3, but it the automatic update doesn’t do the job.

firebug does work in FF3 you just have to go to the Firefox add-ons page and get it from there. For some reason the updater doesn’t do it.