Emerging Order

As we found in several Azure projects, designing complex inter-system interaction is very challenging. Most Azure projects would involve multiple VMs, and multiple components or processes inside each VM, so they are typical inter-system design problems.


SOA (Service Oriented Architecture), although sounds old, is probobaly the most effective way to handle this challenges. However, we need to take a special view through the SOA len. We should emphasis more on Independence. If we can design autonomous and independent components, who can respond to the environment and messages in an intellectural and responsive way, the Order of the inter-system will Emerge as a result.  The complex behavior of the inter-system is hard to be designed, rather it will emerge from the group of independent components.


We learned a lot about the principle of Emerging Order from this project.
http://cloudsamurai.codeplex.com/

Use VSTS to load test Azure based app

How to do load test on Azure based app? Using VSTS in Azure is a natural choice, since both the test rig and servers been tested would be in cloud and be freely scaled out. However, testing with VSTS in Azure today could be painful, mainly due to the network issues. We acturally did such a test recently. Below is our findings.


(The to-be-released new persistent VM role, as well as the improved Azure network capabilities like Brooklyn and internal DNS, will hopefully make the VSTS testing in Auzre much easier in the future.)

• Can we do VSTS load test from Azure and test App in Azure?
• Yes
• Deployment architecture
• (this is the design works for us, but not necessarily the best design.)
• Test Controller in Azure, worker role
• Test Agents in Azure, worker role
• Controller and Agents are in the same worker role, on different instances
• Use Azure connect to group Test Controller and Test Agents so that they can see each other
• VSTS test DB in SQL Azure
• App server (being tested) in Azure, whatever role
• Workarounds and Pains
• VSTS DB doesn’t work directly in SQL Azure
• We used customized DB scripts
• Test Rig (Agent/Controller/TestedServer) is very unstable in Azure
• Controller may lose connection to Agent. Agent may not start.
• If this happens during the test, the test will fail.
• We are forced to use only up to 3 agents.
• Every time this happens, we need to restart the service in Agents or controller.
• Hard to collect performance counters

An open source multi-tenant framework for Azure cloud

We developed a Multi-Tenancy Framework for partners who want to realized multi-tenancy but 1) don’t want to make change on their current single tenant design, or 2) want to keep high level of isolation between tenants, so that to use local cache and allow per-tenant customization, which is only achievable in on-premise environment before. We originally developed this for two of our managed partners. Then, we decided to make it open source, so that everyone can benefit from it.

It is available now at http://cloudsamurai.codeplex.com/ . We just released v0.9 yesterday. Welcome to try and feedback.

How to do QA and test in Azure (4)

·         Auto-deployment Tool is very important

o    Save lots of time, test quickly

o    Confidence in testing environment, focus on features

o    You also gain confidence in the deployment tool, which allows you to deploy more often, and faster.

·         Small Release, More Often Release

o    Integrate customer feedback into bug fixing

o    Drive better customer experience

o    Drive cost-per-release down

o    Release often. How often? One release every month/week/day ?

·         "One week" worked for some web sites.

·         In reality, this should simply be as low as is practical and useful for the business needs of the group.

·         We found that the lower we could get this, the more flexibility we had in getting changes out to prod quickly and reliably, getting better customer experiences faster. 

·         In particular, get really fast at verifying that things that haven't changed aren't broken.

·         This will really help with testing things like adopting new versions of the SDK.

o    More-often-and-quicker release is enabled by

·         Good auto-deployment tools.

·         Well documented deployment and troubleshooting processes.

How to do QA and test in Azure (3)

·         Shared nature of cloud test environment

o    It's not practical for each tester to have his/her own dedicated test environment in cloud.

o    Handle confliction:

·         No destructive test without prior coordination

·         Time management

·         Different cloud env for different build

·         Load test

o    How to generate enough work load

·         Not hard to build a worker role to do this, may be 3rd party solutions. 

·         VSTS Load Test Agent/Controller can be used from on-premises, but you may have network proxies that make this more difficult.

o    Test Metrics: the same as web app load test

·         Most important is learning how many instances, of what size, you will need to handle the user load you're expecting, and still give those users a good experience.

·         Security test

o    All web app security test practices apply to Azure app

o    Assume every access as an attack

o    Everybody can access your Azure app

o    Do a proper threat model, and follow any applicable security best practices.

How to do QA and test in Azure (2)

·         Test in Cloud Best Practice

o    Keep at least two environments in clouds

·         "Last Known Good" environment

§  To be used for testing, stakeholder demos, or perf/load/stress testing

·         "Latest Build" environment

§  May have any build on it, and may or may not work depending on what ever else is going on. 

§  This environment can also be used for any kind of destructive testing, since any test that shouldn't be interrupted should be run in the "LKG" environment.

·         Alternating the environments (Env1 and Env2)

§  In practice, you can get this effect by simply alternating environments that you deploy your build to. 

§  So Env1 gets Build 1, Env2 gets Build 2, then Env1 gets Build 3.  Then, if build 4 fails BVT's in Env2, You'll still have Build 3 in Env 1.  Build 5 should then still go to Env2, and this should continue until a stable build exists in Env2. 

§  Ideally, this means the most recent two builds will always be deployed, unless one build fails.

o    The cloud test environment should be as "real" as possible. 

·         Best case, you use production versions of all dependencies, real certificates, real domain names, and production-identical configuration settings. 

·         This isn't always practical (for example cache durations are frequently lower in test than prod to allow content to expire more rapidly), but the closer you can get to this, the less surprised you'll be when you go to production.

o    Do at least one cloud update each day, or even more often

·         This helps stabilize deployment tools, encourage cloud-based testing (code won't be too stable to test), and get people familiar with what can go wrong in the cloud, and how to fix it.

How to do QA and test in Azure (1)

·         Where to test: Test in Local vs. Test in Cloud

o    Local: fast, cheap, but may not be real

o    Cloud: slow, costly, but real

o    General rule:

·         Do as many test as possible in local

§  If some test can be tested in local, test it in local.

§  If some test can't be tested in local, ask yourself: Can we configure the code to be able to test in local?

·         Do more test in cloud than in local

§  Especially in later phases of project, do more tests in cloud than in local

o    Tests to be done in local

·         Debug/Functional Test

·         UI Automation Test

·         Performance (Profiling only)

o    Test to be done in Cloud

·         Integration

·         Load test: scale-out

·         Failover. Failover will happen at some point: know how your application handles it.

·         Performance (end to end performance, end-user experience)