How to do QA and test in Azure (3)

·         Shared nature of cloud test environment

o    It's not practical for each tester to have his/her own dedicated test environment in cloud.

o    Handle confliction:

·         No destructive test without prior coordination

·         Time management

·         Different cloud env for different build

·         Load test

o    How to generate enough work load

·         Not hard to build a worker role to do this, may be 3rd party solutions. 

·         VSTS Load Test Agent/Controller can be used from on-premises, but you may have network proxies that make this more difficult.

o    Test Metrics: the same as web app load test

·         Most important is learning how many instances, of what size, you will need to handle the user load you're expecting, and still give those users a good experience.

·         Security test

o    All web app security test practices apply to Azure app

o    Assume every access as an attack

o    Everybody can access your Azure app

o    Do a proper threat model, and follow any applicable security best practices.

How to do QA and test in Azure (2)

·         Test in Cloud Best Practice

o    Keep at least two environments in clouds

·         "Last Known Good" environment

§  To be used for testing, stakeholder demos, or perf/load/stress testing

·         "Latest Build" environment

§  May have any build on it, and may or may not work depending on what ever else is going on. 

§  This environment can also be used for any kind of destructive testing, since any test that shouldn't be interrupted should be run in the "LKG" environment.

·         Alternating the environments (Env1 and Env2)

§  In practice, you can get this effect by simply alternating environments that you deploy your build to. 

§  So Env1 gets Build 1, Env2 gets Build 2, then Env1 gets Build 3.  Then, if build 4 fails BVT's in Env2, You'll still have Build 3 in Env 1.  Build 5 should then still go to Env2, and this should continue until a stable build exists in Env2. 

§  Ideally, this means the most recent two builds will always be deployed, unless one build fails.

o    The cloud test environment should be as "real" as possible. 

·         Best case, you use production versions of all dependencies, real certificates, real domain names, and production-identical configuration settings. 

·         This isn't always practical (for example cache durations are frequently lower in test than prod to allow content to expire more rapidly), but the closer you can get to this, the less surprised you'll be when you go to production.

o    Do at least one cloud update each day, or even more often

·         This helps stabilize deployment tools, encourage cloud-based testing (code won't be too stable to test), and get people familiar with what can go wrong in the cloud, and how to fix it.

How to do QA and test in Azure (1)

·         Where to test: Test in Local vs. Test in Cloud

o    Local: fast, cheap, but may not be real

o    Cloud: slow, costly, but real

o    General rule:

·         Do as many test as possible in local

§  If some test can be tested in local, test it in local.

§  If some test can't be tested in local, ask yourself: Can we configure the code to be able to test in local?

·         Do more test in cloud than in local

§  Especially in later phases of project, do more tests in cloud than in local

o    Tests to be done in local

·         Debug/Functional Test

·         UI Automation Test

·         Performance (Profiling only)

o    Test to be done in Cloud

·         Integration

·         Load test: scale-out

·         Failover. Failover will happen at some point: know how your application handles it.

·         Performance (end to end performance, end-user experience)