How to do QA and test in Azure (4)

·         Auto-deployment Tool is very important

o    Save lots of time, test quickly

o    Confidence in testing environment, focus on features

o    You also gain confidence in the deployment tool, which allows you to deploy more often, and faster.

·         Small Release, More Often Release

o    Integrate customer feedback into bug fixing

o    Drive better customer experience

o    Drive cost-per-release down

o    Release often. How often? One release every month/week/day ?

·         "One week" worked for some web sites.

·         In reality, this should simply be as low as is practical and useful for the business needs of the group.

·         We found that the lower we could get this, the more flexibility we had in getting changes out to prod quickly and reliably, getting better customer experiences faster. 

·         In particular, get really fast at verifying that things that haven't changed aren't broken.

·         This will really help with testing things like adopting new versions of the SDK.

o    More-often-and-quicker release is enabled by

·         Good auto-deployment tools.

·         Well documented deployment and troubleshooting processes.

How to do QA and test in Azure (3)

·         Shared nature of cloud test environment

o    It's not practical for each tester to have his/her own dedicated test environment in cloud.

o    Handle confliction:

·         No destructive test without prior coordination

·         Time management

·         Different cloud env for different build

·         Load test

o    How to generate enough work load

·         Not hard to build a worker role to do this, may be 3rd party solutions. 

·         VSTS Load Test Agent/Controller can be used from on-premises, but you may have network proxies that make this more difficult.

o    Test Metrics: the same as web app load test

·         Most important is learning how many instances, of what size, you will need to handle the user load you're expecting, and still give those users a good experience.

·         Security test

o    All web app security test practices apply to Azure app

o    Assume every access as an attack

o    Everybody can access your Azure app

o    Do a proper threat model, and follow any applicable security best practices.

How to do QA and test in Azure (1)

·         Where to test: Test in Local vs. Test in Cloud

o    Local: fast, cheap, but may not be real

o    Cloud: slow, costly, but real

o    General rule:

·         Do as many test as possible in local

§  If some test can be tested in local, test it in local.

§  If some test can't be tested in local, ask yourself: Can we configure the code to be able to test in local?

·         Do more test in cloud than in local

§  Especially in later phases of project, do more tests in cloud than in local

o    Tests to be done in local

·         Debug/Functional Test

·         UI Automation Test

·         Performance (Profiling only)

o    Test to be done in Cloud

·         Integration

·         Load test: scale-out

·         Failover. Failover will happen at some point: know how your application handles it.

·         Performance (end to end performance, end-user experience)