An open source multi-tenant framework for Azure cloud

We developed a Multi-Tenancy Framework for partners who want to realized multi-tenancy but 1) don’t want to make change on their current single tenant design, or 2) want to keep high level of isolation between tenants, so that to use local cache and allow per-tenant customization, which is only achievable in on-premise environment before. We originally developed this for two of our managed partners. Then, we decided to make it open source, so that everyone can benefit from it.

It is available now at http://cloudsamurai.codeplex.com/ . We just released v0.9 yesterday. Welcome to try and feedback.

How to do QA and test in Azure (4)

·         Auto-deployment Tool is very important

o    Save lots of time, test quickly

o    Confidence in testing environment, focus on features

o    You also gain confidence in the deployment tool, which allows you to deploy more often, and faster.

·         Small Release, More Often Release

o    Integrate customer feedback into bug fixing

o    Drive better customer experience

o    Drive cost-per-release down

o    Release often. How often? One release every month/week/day ?

·         "One week" worked for some web sites.

·         In reality, this should simply be as low as is practical and useful for the business needs of the group.

·         We found that the lower we could get this, the more flexibility we had in getting changes out to prod quickly and reliably, getting better customer experiences faster. 

·         In particular, get really fast at verifying that things that haven't changed aren't broken.

·         This will really help with testing things like adopting new versions of the SDK.

o    More-often-and-quicker release is enabled by

·         Good auto-deployment tools.

·         Well documented deployment and troubleshooting processes.

How to do QA and test in Azure (3)

·         Shared nature of cloud test environment

o    It's not practical for each tester to have his/her own dedicated test environment in cloud.

o    Handle confliction:

·         No destructive test without prior coordination

·         Time management

·         Different cloud env for different build

·         Load test

o    How to generate enough work load

·         Not hard to build a worker role to do this, may be 3rd party solutions. 

·         VSTS Load Test Agent/Controller can be used from on-premises, but you may have network proxies that make this more difficult.

o    Test Metrics: the same as web app load test

·         Most important is learning how many instances, of what size, you will need to handle the user load you're expecting, and still give those users a good experience.

·         Security test

o    All web app security test practices apply to Azure app

o    Assume every access as an attack

o    Everybody can access your Azure app

o    Do a proper threat model, and follow any applicable security best practices.

How to do QA and test in Azure (2)

·         Test in Cloud Best Practice

o    Keep at least two environments in clouds

·         "Last Known Good" environment

§  To be used for testing, stakeholder demos, or perf/load/stress testing

·         "Latest Build" environment

§  May have any build on it, and may or may not work depending on what ever else is going on. 

§  This environment can also be used for any kind of destructive testing, since any test that shouldn't be interrupted should be run in the "LKG" environment.

·         Alternating the environments (Env1 and Env2)

§  In practice, you can get this effect by simply alternating environments that you deploy your build to. 

§  So Env1 gets Build 1, Env2 gets Build 2, then Env1 gets Build 3.  Then, if build 4 fails BVT's in Env2, You'll still have Build 3 in Env 1.  Build 5 should then still go to Env2, and this should continue until a stable build exists in Env2. 

§  Ideally, this means the most recent two builds will always be deployed, unless one build fails.

o    The cloud test environment should be as "real" as possible. 

·         Best case, you use production versions of all dependencies, real certificates, real domain names, and production-identical configuration settings. 

·         This isn't always practical (for example cache durations are frequently lower in test than prod to allow content to expire more rapidly), but the closer you can get to this, the less surprised you'll be when you go to production.

o    Do at least one cloud update each day, or even more often

·         This helps stabilize deployment tools, encourage cloud-based testing (code won't be too stable to test), and get people familiar with what can go wrong in the cloud, and how to fix it.

How to do QA and test in Azure (1)

·         Where to test: Test in Local vs. Test in Cloud

o    Local: fast, cheap, but may not be real

o    Cloud: slow, costly, but real

o    General rule:

·         Do as many test as possible in local

§  If some test can be tested in local, test it in local.

§  If some test can't be tested in local, ask yourself: Can we configure the code to be able to test in local?

·         Do more test in cloud than in local

§  Especially in later phases of project, do more tests in cloud than in local

o    Tests to be done in local

·         Debug/Functional Test

·         UI Automation Test

·         Performance (Profiling only)

o    Test to be done in Cloud

·         Integration

·         Load test: scale-out

·         Failover. Failover will happen at some point: know how your application handles it.

·         Performance (end to end performance, end-user experience)

The Essence of Multi-Tenancy

The Essence of Multi-Tenancy is Share Resource:

• Share resource -> Share cost -> Low Cost per Tenant
• Car model -> Bus model

There are two forces impact multi-tenancy design: Standardization & Isolation. We must achieve a balance between these two forces based on our particular situation, and achieve a right level of multi-tenancy design.

There are several related issues to multi-tenancy design:

• Multi-Tenancy and Scale-Out
• Multi-Tenancy: many small tenants share one computing resource
• Scale-Out: one big tenants runs on a cluster of computing resources
• Multi-Tenancy + Scale-Out: many tenants share a pool of clustered computers
• Multi-Tenancy and Customization
• Multi-Tenancy: sharing everything, similarity
• Customization: flexibility, tailor to unique need
• Multi-Tenancy + Customization: Balance

Cloud Computing Notes

About me:

I'm a Developer Evangelist in Microsoft, selling cloud computing technologies to our global software partners.

Personally, I believe in cloud computing and I'm confident in Microsoft cloud computing platform Azure. I had to admit that Azure is not perfect: many features are still missing or immature, and the developer experience on Azure is some times painful. So, I blog here to share experience we gained from real world Azure projects. Some experience may be applied to other cloud computing platforms as well.

In my spare time, I participate in open source projects: http://cloudsamurai.codeplex.com/ and http://cloudninja.codeplex.com/, both of which are about how to use Azure to build cloud applications.

In a sense, I'm lucky that I'm doing a job I like, and I'm selling things I really believe to the people I love.