Architecture Update

The SQL Server 2012 version of PDW introduces some major architecture changes:

• As show in the picture below the Landing Zone and the Backup node have been removed from the appliance and there good reasons for. The standard sizing of these component didn’t met most of the customer requirements and it was very hard to find a configuration that meets 80% of PDW customers. So decision has been made to remove these components and customers have now more flexibilities to size the ETL and the Backup node to their needs.

• Expensive storage components have been replace by an economical high density Direct Attached Storage
• Virtualization of the Software Components based on Windows Server 2012 and Hyper-V
• Modular Design and Smarter Scale Out: SQL Server 2012 PDW has reshaped the very hardware specifications required of an appliance through innovations from the software to deliver optimal value to customers. The new version introduces a new modular design, as show in the following picture:

The smallest PDW appliance consists of a Base Scale Unit (with a Passive Scale Unit for HA) and can be extended by a number of Capacity Units. If you reach the physical space limit of the rack you can add up to 6 more racks.

Depending of the vendor the units are shipped differently: HP offers a Base Scale Unit with 2 compute nodes and every Capacity Unit is also shipped with 2 compute nodes (left side of the picture). Dell offers the Base Scale Unit and the Capacity Unit with 3 compute nodes each (right side of the picture).

If we have a look at the capacity the smallest PDW appliance from HP (Quarter Rack with only the Base Scale Unit) offers a capacity of 53 – 227 TB (depending on the compression rates) and a raw disk space of 45 TB (with 3 TB disks). The Quarter Rack of DELL provides a capacity of 79 – 3470 TB (depending on the compression rates) and a raw disk space of 68 TB (with 3 TB disks).

• Further Scale: Based on the new modular hardware design and a hardware refresh PDW offers now a scale out capacity up to 5 Petabytes.

Software Architecture Overview

The new version of PDW provides also some major software updates. As the product name already states PDW is now running on SQL Server 2012. The operating system on all hosts is Windows Server 2012 Standard edition. All fabric and workload activity happens in Hyper-V virtual machines which also run on Windows Server 2012 Standard edition. A PDW Agent runs on all hosts and all VMs and collects appliance health data on fabric and workload.

A special PDW version of SQL Server 2012 Enterprise Edition is used on the Control node and on all Compute nodes to provide high scale database capabilities.

New Features

Additionally to the new hardware and software architecture I want to highlight some more very interesting features for customers:

• Columnar Storage: Microsoft continues the rollout of xVelocity and provides with SQL Server 2012 PDW a new primary storage type for databases. Customers can now choose between a row store and a new updateable version of the xVelocity memory optimized columnstore as table storage format. This means that we can define a writable Clustered Columnstore Index (Updates and bulk load are fully supported) at a table so that the whole table is stored into memory and we benefit of a much higher compression by the column oriented storage format.
• Visual Studio 2012 Integration: As you probably know the tool used for database administration and database development in PDW Version 1 was Nexus. With the new version of PDW we have now full support for SQL Server Data Tools for Visual Studio 2012.

Also the project types for SSIS, SSRS & SSAS are fully supported with Visual Studio 2012.

• Monitoring Enhancements: The whole monitoring of sessions, queries, loads, appliance health status and performance information has been completely redesign. Microsoft did a very good job here and the design looks like the Azure Portal. This means also for customers that monitoring and operations will look like the same On Premises as well as in the Cloud on Windows Azure. The following screenshot gives an impression of the new web user interface:

• Polybase & Hadoop Integration: Today almost every big database vendor also provide a “Big Data” solutions based on Hadoop. Whether it’s a vendor specific distribution like IBM’s BigInsights solution, which is based on Cloudera or Microsoft’s specific implementation of Hortonworks Hadoop distribution called HDInsight or a full appliance which comes pre-installed with Hadoop. No matter which Hadoop platform you choose you still have the challenge of integration. Some type of data and analytics will happen on Hadoop (like text, log or sensor analysis) but we will still use databases for BI & Reporting.

While this might be something you have decided to do, you realize that there is a big learning curve when your IT department needs to re-orient themselves around HDFS, MapReduce, Hive, Hbase, etc. rather than T-SQL and a standard RDBSMS design. It will require a significant re-training around Hadoop and the ecosystem as well as a major effort to integrate the Hadoop implementation with the data warehouse.

In order to meet the requirements of a modern data platform, it must provide insights to your end users without having to acquire another tool from a third party or another expensive appliance offering to purchase.

The unfortunate reality is that no one vendor can deliver on all the options you need at a cost that you want to pay. They either have a data warehouse solution but no BI or provide BI but no data warehousing. Some vendors provide a Big Data solution but is disconnected with their data warehouse solution. Finally, some vendors might have a solution for each workload and will happily charge you millions of euros or dollars for them all.



Microsoft goes a different way and brings with SQL Server 2012 PDW an integrated query layer called “Polybase” which enables queries across Hadoop and SQL Server. Data structures stored in Hadoop are described by tables in PDW and customers can consume these data by standard T-SQL and can also join those tables with normal relational ones. So to end users that only consume this data it’s totally transparent where the data comes from and IT departments can use their normal database skillset to work with Hadoop. How does those kind of tables look like?



Now we can easily query and join this table.

Polybase is developed by the PDW team together with the Gray System Lab and its famous team lead David DeWitt. More detailed information can be found on the project page: http://gsl.azurewebsites.net/Projects/Polybase.aspx or in the video of the PASS 2012 conference http://www.sqlpass.org/summit/2012/DavidDewittSpotlight.aspx

Summary

As you can see from this post Microsoft did very heavy investments in its modern Big Data platform and gives customers the possibility to invest build high scale solutions on SQL Server 2012 PDW as well as on HDInsight for Hadoop based workloads. With Polybase customers get a fully integrated Hadoop query engine into the Data Warehouse Layer that can easily consumed with T-SQL knowledge.

The new architecture and the modular design of the appliance gives customers the possibility to start with small investments and scale very cost efficient on demand.

In my next post I will talk in more detail about a POC I did with the new version and the results and lessons learned.

Source: Microsoft slide decks of SQL Server 2012 Parallel Data Warehouse

I had the chance to do a customer POC on a very early version of SQL Server 2012 Parallel Data Warehouse and like to share the new features for PDW as well as the impressive results we got. The good thing with this POC is, that we did the same POC also on the SQL Server 2008R2 version of PDW so we have a direct comparison on the results.

If you are not familiar with Parallel Data Warehouse I can suggest my further post list about PDW to get a step into it: http://dwjunkie.wordpress.com/pdw/

I will split this post into several topics that I will explain in more details in my next posts:

• What’s new in SQL Server 2012 PDW?
• Scope & Customer Expectations of the POC
• SQL Server 2008 R2 PDW vs. SQL Server 2012 PDW
• Performance Results and lessons learned (SSIS vs. TSQL, Querying, Cube Processing, Compression, Limitations, etc.)

Source: http://www.worldwithlove.org/blog/lets-rock-the-green-music-festival/

In the second part of the guide I will describe the configuration scenario in general, recommended default configurations, how to setup parameters and environments and migration options for existing SQL Server based configurations.

Configuration Scenario

As mentioned in my first post in previous versions of SSIS 2012 we had several possibilities how to configure our SSIS packages. With new concept of parameters and environments in SSIS 2012 the way to use configurations is now standardized, as shown in the picture below.

If you follow the design of master and child packages all configurable parameters are now stored in one environment, that is stored in the SSIS catalog. A SQL Server agent job, which triggers the master package of your ETL process, has an environment assigned, which will be used for package execution. All child packages can then use the parameters of the assigned environment.

This also means that the old concept of assigning different configurations also to child packages is not possible anymore. Shared configuration parameters used by several packages should be defined on a project level. If you want to share parameters across projects you need to stored them redundant in different environments and manage it on your own. I know that this is a small loose of flexibility but it is currently the only way to go.

Default Configurations

The following list contains recommended SSIS properties which should be made dynamic configurable by default in every SSIS project / package:

In order to deploy your project you just right-click on your project and press click “Deploy”, which will open the deployment wizard where you need to specify the SSIS server name and the target path, like in my sample below.

After you are done with the wizard you should see your project and all containing SSIS packages in the SSIS catalog.

Environment Setup

If you look carefully to your “Environments” folder you will recognize that it’s empty. In order to configure your project you need to create an environment first. Typically you will then add all defined parameters to your project environment. In larger projects this can take some time, if you do it manually. In order to make it a little bit more comfortable you can also use this stored procedure that analyzes all defined parameters of a project in the SSIS catalog (from the “internal.object_parameters” table in the SSISDB) and adds them to your new environment.

USE [SSISDB]
GO

DECLARE @RC int
DECLARE @environment nvarchar(128)
DECLARE @folder nvarchar(128)
DECLARE @project nvarchar(128)

-- TODO: Set parameter values here.

EXECUTE @RC = [dbo].[CreateEnvironmentFromParameters] 
   'MyEnvironmnent' -- name of the environment
  ,'Test' -- project folder 
  ,'Configuration_SQL2012' -- project name
GO

This and all following stored procedures can be found here:

https://skydrive.live.com/redir?resid=EDD20A747F7B8FDE!32835&authkey=!AFvRzY5jzfphR_Y

“CreateEnvironmentFromParameters_SP.sql”

The stored procedure has four input parameters, each one is mandatory:

General + _ + ParameterName
PackageName + _ + ParameterName

• XML configuration files
• Environment variables
• Registry entries
• Parent package variables
• SQL Server configurations

In SSIS 2012 the configuration handling has been totally redesigned and there is now one common standard way how to do this.

This guide will show how to use the new way of configuration, how parameters and environments can be used to build dynamic SSIS packages, how they simplify the deployment process and how to migrate old configurations to the new world.

A special thanks goes to my colleague Christian Jurjut, who wrote this guide with me.

Introduction

In SSIS 2012 parameters on project and package level have been introduced. In combination with the also new “environments” in the SSIS catalog, they offer a new opportunity to configure packages during runtime. This new method was introduced in order to have only a single way to apply configurations and so to eliminate the bugs and weaknesses of the old configurations.

Deployment Models

The deployment model helps to deploy SSIS packages and projects to different environments, like a development, test and production system. Therefore it is possible to provide different configurations for each system already in the solution during design time, also to be able to test packages with different parameter settings. In order to provide those configurations it is necessary to create different scenarios in the project’s configuration manager. It is reachable under the project properties.

In the appearing window it is possible to create new configuration scenarios by selecting “<New…>” from the “Active solution configuration” drop down field.

Usually the “Server Name” and the “Server Project Path” differ between the environments. So by setting those values it is possible to deploy the SSIS packages to different systems only by choosing a value from a drop down field.

The deployment model is not completely new to SSIS projects, but with SSIS 2012 a new model was introduced to the already existing package deployment model. It is called “project deployment model” and it is mandatory to create projects using this deployment model in order to be able to use parameters and deploy SSIS packages to the SSIS catalog. The “old” package deployment model can still be used but it is not possible to apply the new configuration methods, as it does not support parameters or the SSIS catalog.

In combination with parameters the project deployment model gets more useful, as it is possible to assign different values to parameters for each scenario. So, for example, it is possible to create a scenario for each environment, like development, test and production, and to assign different connection strings to those scenarios via parameters. This gives you the opportunity to execute SSIS packages against different systems during design time only by choosing another scenario from the drop down box. But it also offers the possibility to deploy the project and packages with already the right configuration to the different environments.

Parameters

Parameters are newly introduced to SSIS with SQL Server 2012. They can be created on project or package level, according to the assigned scope. If a parameter is defined on project level, it can be used in all SSIS packages within this project. If it is created on package level, the parameter can only be accessed within the according package.

Parameters can be used like variables, with a few exceptions. It is possible to assign values to them and to use them within expressions for different components. So they can be used for configuration purposes. Unfortunately it is not possible to replace variables at all by parameters, as they cannot store results from “Execute SQL Tasks”.

On project level parameters can be created by using the “Project.params” entry in the solution explorer.

By clicking the “Add Parameter” button, new parameters will be added to the list below. The following properties can be provided for them:

• The name of the parameter
• The according data type
• The default value, used during design time
• A sensitive flag
• A required flag
• And a description

If the parameter is marked as sensitive, its value will be encrypted when the project is deployed to the SSIS catalog. The required flag indicates that it is mandatory to pass a value to this parameter before the package can be executed.

To create parameters at package scope, you just have to choose the “Parameters” tab in the according package, where you can also add them by using the “Add parameter” button.

If different scenarios have been applied to the project, it is possible to assign different values to parameters for each scenario. This can be achieved by clicking the “Add Parameters to Configurations” button. The following window gives an overview of all available scenarios (columns) and the configured parameters (rows).

By clicking the “Add” button a list of all available parameters appears, which can be added to the configuration.

Integration Services Catalog

The SSIS catalog is new to SQL Server 2012 and it is used to store, manage and monitor deployed packages. The SSIS repository is now an integrated part of SQL Server and the old SSIS Service is only there for compatibility issues. This also means that we can now easily backup and restore the whole repository as a database and all the packages, configuration settings and logging information is included.

In order to be able to deploy SSIS packages to SQL Server, the integration services catalog has to be created on the server. The catalog uses CLR stored procedures, so you will need to enable CLR integration on the database engine, if it is not enabled already. After CLR integration has been enabled, you can use SQL Server Management Studio Object Explorer in order to create the catalog. As shown in the following image, right-click the Integration Services node and then click “Create Catalog”.

In the create catalog dialog box you will be asked to supply a password. This password is used to create a database master key, which will be used to encrypt sensitive data in the catalog. The master key has to be backed up separately from the database. So it is recommended to create a backup as soon as it is created. Therefore the master key has to be decrypted and an encrypted backup has to be created. The following script can be used for this purpose:

OPEN MASTER KEY DECRYPTION BY PASSWORD = 'password'; 

BACKUP MASTER KEY TO FILE = 'c:\temp\exportedmasterkey' 

ENCRYPTION BY PASSWORD = 'new password'; 

After creation the object explorer displays the catalog twice, first in the databases node and then in the Integration Services node. In the databases node, you can work with the catalog, SSISDB, like any other user database. In the integration services node, you can administer the catalog by right-clicking the corresponding child nodes.  

Within the catalog projects and packages are stored in folders. On each folder it is possible to delegate user rights in order to permit administration. The catalog also stores all environments which will be used later to configure packages during runtime. So each folder, created in the SSIS catalog contains two subfolders. The “Projects” folder, where SSIS packages will be deployed and stored, and the “Environments” folder, which manages the different environments, created for certain projects.

After deploying and executing SSIS packages the integration services catalog provides information regarding versioning, execution results and monitoring. The information can be retrieved by using the provided standard reports or by querying the SSISDB. For this reason some example queries are supplied here:

https://skydrive.live.com/redir?resid=EDD20A747F7B8FDE!32835&authkey=!AFvRzY5jzfphR_Y

• The following query selects the execution events and messages created by SSIS package execution: “Check Execution Events and Messages.sql”
• The following query provides the runtime duration for each package executed: “Check Package Execution and Duration Per Package.sql”
• The following query provides the duration for each task executed within SSIS packages: “Check Package Execution and Duration Per Task.sql”
• The last query combines the event messages with the execution duration: “Check Execution Messages and Durations.sql”

Environments

In SSIS 2012 environments in the SSIS catalog can be used to assign different configurations to deployed projects and packages during runtime. After the SSIS catalog has been created and configured and at least one custom folder has been created, in order to store projects and packages, environments can be created in the according subfolder. Each custom folder can store different projects and environments. So it is possible to reuse already defined environments in certain projects as well as to define more than one environment per project. So, for example, you can define a delta and an initial environment for your projects, if your packages can handle both.

An environment can be created by right-clicking the “Environments” folder and selecting “Create Environment” from the context menu. In the first step you can only name the new environment. Now a new entry appears in the “Environments” folder. When you open it a new window appears, which provides some general information and the ability to assign variables and permissions.

Variables have the same properties as parameters, except the “required” option. In the example above two variables were used to store the source and destination connection string and one variable, “DeltaLoad” is a used as indicator either the package is executed in delta or in initial mode.

By adding the environment as reference to a deployed project, it can be used to configure parameters and connections.

Information regarding environments can also be retrieved from the SSISDB. It is stored in the following four tables:

• internal.environments: Stores the unique id, the name and the folder in which the environment is located.
• internal.environment_variables: Provides information about variables used in environments, like the name, data type and the current value.
• internal.environment_references: Stores the relationship between projects and environments. It provides the information which project references which environment.
• internal.environment_permissions: Stores information about permissions for each environment.

These tables also offer the possibility to modify existing environments or to create new ones. So, for example, they can be used to migrate the “SSIS Configurations” table to new environments. How this can be done will be explained later on.

Package and project configuration

After a project has been deployed to the SSIS catalog, it is possible to configure it by right-clicking the project or one of the package entries and choosing “Configure…” from the context menu. The arising window manages all parameters and connection managers on project and on package scope.

If a project contains many packages and therefore many parameters, the default layout might be confusing, because it contains all available parameters and connection managers. But the current scope can easily be changed by choosing another value from the according drop down field.

For parameters the following information is provided:

• the container determines either the parameter is on project level or belongs to a package
• the name of the parameter
• its current value

For connection managers the following information is provided:

• the connection string of the connection manager
• the initial catalog and the server name as single values
• user name and password for authentication purposes
• the “RetainSameConnection” property

The value of these properties can by changed here, so that the changes take effect on the project and the corresponding packages.

Under the reference section it is possible to assign environments to the project, which can be used to configure the values of the different properties. The assignment of the environmental variables to parameters or connection manager properties can be done in the according section by choosing the “…” button next to each of these values.

In the upcoming “Set Parameter Value” window you can choose between, editing the value manually, using the default value or assigning a variable from the used environments.

For each parameter, which has to be configurable, one variable has to be created in the according environments. So, for example, if you want to dynamically configure which packages are executed in initial or delta mode, you have to create one variable per package. If you would create only one variable for the whole project, all packages would be executed either in initial or in delta mode.

Contrary to this, if more than one environment is used within a project and each of them should configure the same parameters and connection managers, it is mandatory that the variable names are equal in all environments. Otherwise the assignment cannot be performed uniquely.

Information about parameters can be also retrieved from the SSISDB table “internal.object_parameters”. It provides information about the project, to which the parameter belongs, the scope the parameter was created at, the used data type and the default value at design time.

Summary

This first chapter introduced the new configuration possibilities in SSIS 2012 and how to use them. Within the next part I will show some Best Practices which parameters you should always make configurable, how to create environment more efficiently and how to migrate existing configuration for older SSIS version to the new SSIS catalog.

I also submitted two sessions, so please vote, if you are interested:

• Germans favour – Microsoft BI on top of SAP
• Build more efficient Data Warehouses via trial and error?

See you @SQLBits…

Functionality

The new version will be based on SQL Server 2012 and will also include the enhancements of the column store index. This means Microsoft ships also in-memory technology within it’s product. Most of the T-SQL limitations that we know from version 1 will be also gone.

Architecture & Hardware

Additionally the new version will have some architectural changes, a hardware refresh (HP EDW v2) and a new Quarter Rack, which is more attractive also for smaller Data Warehouse projects and customers.

Big Data & more

As you probably also heard from the PASS Summit PDW Version 2 will ship with Polybase. Polybase provides users the ability to query against structured data that is stored in PDW and unstructured data that is stored in Hadoop in the same way (T-SQL).

This means that Parallel Data Warehouse is and will be Microsoft’s strategic Big Data Platform and Appliance that combines the scale and performance of a relational Data Warehouse with the velocity, variety and volume of unstructured data stored in Hadoop.  And as Google already figured out NoSQL will not be the solution to handle all kind of data in a way that query performance will be acceptable by business users. If there is a certain structure in your data, you probably want to use it.

If you are interested in more details about Polybase have a look at the session of David DeWitt: http://www.sqlpass.org/summit/2012/DavidDewittSpotlight.aspx

I also took the chance to visit the PASS Conference and was very impressed about this fast growing community. I remember when I was there in 2009 the room of the key not was only have of the size is has been now, that was really fascinating. Along with all the new stuff like Hekaton, writable Column Index Store, PDW Version 2 and all the other new things Microsoft has been shown us I believe we will have a fantastic year 2013 with lot’s of exciting projects. So it’s time for me to catch up here.

ETL Performance

As shown in my series “SSIS Design for Performance and Scale” a typically designed SSIS package with only 1 stream in the dataflow will give us a write performance of around 25 MB/s. Because data load performance is a critical issue in the current customer environment one of the original SSIS packages looked like this:

What we can see here are 6 sequence container that are executed as 6 threads in parallel, every container handles a subset of the data. But there are several disadvantages with that kind of design:

• Complex parallel loads of partitions (partition by quarter)
• Manual parallelism within package design
• Multiple implementation of business logic per thread

With PDW we wanted to see how much the engine itself can help us in order to parallelize data loads. So we decided to remove all parallelism from the package so that it looks like this:

This gives us the following advantages:

• Parallelism of load processes is handled by PDW (let’s see)
• Much simpler SSIS Design -> less implementation effort
• Business logic exists only once -> simpler and faster maintainability

So far so good, but what about the performance numbers? Let’s have a quick look at the 2 scenarios we have tested. We used the Landing Zone as host for Integration Services and executed all packages on this server.

Scenario 1

In the original customer scenario the SSIS package loads data from on table to another table within the same database. The whole database is stored in PDW.

Scenario 2

Because we had some trouble with the read performance we decided to put a backup of the database to the landing zone and load the data from the landing zone to the PDW nodes.

Performance Results

The following matrix shows the performance numbers we got during our tests.

The column “Production Duration” shows the reference value from the current customer environment. This is our baseline number.

Now we can see 2 interesting behaviors. The first scenario is 2x slower, but the second one up to 4x faster. So what is the reason for these big differences in performance?

• Read data from PDW is slow, around 8 MB/s (OleDB driver) –> Scenario 1
• Load data into PDW is fast, around 45 MB/s –> Scenario 2
• PDW automatically distributes loading processes

Important: The product group is aware of the slow OleDB driver and is working hard to make it faster. We have already seen a much better performance on a later Appliance Update.

Summary

I always hear people saying that SSIS is not the right tool for data loading scenarios with PDW. It’s maybe not the fastest option, but we have seen nevertheless good performance measures. Just to remember the easiest SSIS package on a normal SQL Server gives us a write performance of 25 MB/s. With PDW we get 45MB/s, so performance is nearly doubled. If this performance is good enough for you, SSIS is still a good choice and of course additionally you can apply the same parallelism techniques for further scale. And sometimes you don’t have the option to migrate all existing SSIS packages to a T-SQL based ETL.

My feeling is, that there is much more possible in case of performance, if the driver becomes faster and more efficient in the future.

Very often also the BCP tool (bulk copy utility) is recommended for data loading. This is a very fast option if you get your source data as flat file. But when you first need to create flat files before you can use BCP you better use SSIS for direct connectivity.

Relational Query Performance

Now let’s have a look at the relational query part. We got 3 reference queries from the customer that we used for our tests. In order to make the queries work with the PDW database we only had to change the schema name (because only the “dbo” schema is currently supported) and that’s it. And the performance? Awesome, as you can see here:

The column “Production Duration” shows the reference value from the current customer environment. This is our baseline number.

All queries are up to 8 times faster. This is the real power that PDW offers us. A highly scalable SQL engine with lot’s of query power. So no surprises here, it works as we had expect it.

Summary

So what do we learn from this POC. I’d like to address the most important things we have seen again in a short bullet list.

Usability

• PDW is not (yet) a normal SQL Server. So you need to understand the architecture of a MPP system and why it works differently to a SMP system
• It’s not a no brainer. You need to think about, why would you need to buy a MPP System and which problem should it solve. It’s not a “one size fit’s all” thing.
• Plan for migration effort. There are features in SQL Server that are currently not supported in PDW. Work closely with the Microsoft team to understand your specific case and there are already a lot of good design pattern you can use to overcome the missing features.

Performance

• The relational query engine is fast. You have seen the results, but you can also write statements that do not well perform. Think about data distribution and other techniques and avoid data shuffling.
• Plan your ETL. As we have see we get good parallelism support for SSIS, although the read problem has to be solved. So the system helps us with built-in parallelism support and we can keep our SSIS packages much simpler for a longer time.
• Think about ELT.  I think in a MPP environment the SSIS engine itself becomes the bottleneck. We take out the data from a highly parallelized engine pump it through a buffer oriented SSIS engine and put it back to our highly parallelized engine. This works like a funnel which scales only to a certain point. If you can start from scratch or optimize certain packages try to use SSIS to bring the data into PDW and do the transformations within the engine.

You are free to draw your own conclusion out of the presented results or you can also contact we for further questions or feedback. Don’t hesitate, just let me know.

References

• Parallel Data Warehouse – Overview
• Parallel Data Warehouse – Architecture Components
• Parallel Data Warehouse – Hardware Components
• Parallel Data Warehouse – POC lessons learned Part 1
• Parallel Data Warehouse – POC lessons learned Part 2

Scale map

The map is divided into partition strategies and insert strategies. I will start with functional partitioned tables, since a single partition behaves similar to a non-partitioned table.

Direct inserts into single partition

The focus of this design pattern is to achieve the maximum performance when loading into a single partition / table. The target table is partitioned by month and I load with multiple streams directly into one partition (heap and clustered Index). The design looks like this:

The data is split equally into multiple streams within the dataflow and I tested 4, 8 and 16 streams in parallel.  In order to split the data I use a modulo function within the “conditional split” task:

DAY(L_SHIPDATE) % 4 == 0

If you need more then 4 destinations you have to add more dataflows to your package because the data source is limited with ~200MB/s. So a good balance between source and destination is 1:4.

Let’s have a look at the performance numbers:

As you can see I got a nearly linear performance scale when loading into a heap table with no fragmentation (blue line). What’s interesting is, that even for a clustered index table I have a linear scale with no fragmentation, if it is a minimal logged operation (orange line)! The difference in case of speed is the sort that I have to pay for a clustered index.

If we are not minimal logged, we see no performance increase but the fragmentation becomes higher and higher.

Choose the right distribution key

In order to create multiple data streams I use a modulo function as mentioned before. But be careful which column you choose for data distribution. In case of a clustered index table the distribution column has to match the clustered index column. You can see the performance differences below:

Why? Because if the data distribution is not align with the clustered index the parallel bulk inserts hit the same pages and are blocked by each other. You can see this behavior in the following picture with lots of resource locks on the same pages.

Sorting on a heap

During investigations of the execution plans I also saw a sort for a heap load.

This sort appears because the heap table is partitioned. And there is nothing we can do about, even pre-sorting the data by partition key does not help.

Conclusion

As a summary the main important things you should remember:

• Loading with multiple streams into a heap table is the fastest option
• Loading into a clustered index is acceptable if it is minimally logged and ranges are not overlapping (hard to achieve).
• The SSIS data source is limited with ~200MB/s
• A good balance of sources & destinations is 1:4. If you need more, create more dataflows.
• Choose the right data distribution key

Partition switch – strategy to avoid sort

The next pattern I want to take a closer look is partition switch. The test case itself looks like this:

• Target table is partitioned by month
• Load with 1 stream into staging table
• Switch staging table into target heap or clustered index table
• 1, 8, 16 month in parallel

The corresponding SSIS package looks like this. Every data flow is responsible for one month of data. The T-SQL tasks handles the partition switch command.

So let’s see what performance we get out of this pattern. The two pictures below show the overall duration in seconds and the write performance in rows per second. The time I measured was the load time from the beginning till data was available in the partition table, which means that the switch operation is included.

The blue line shows the performance of a heap table and the red one of clustered index load. Just for comparison the blue dotted lines show the performance of a heap load with direct inserts. I compare on moth of data to a direct insert with 1 stream and 8 month of data to 8 parallel streams and so on.

Conclusion

Compared to the first design pattern, what do we learn from this one:

• As we can see partition switch is faster then direct inserts, because we avoid the sort by partition key now
• The clustered index load is a bit slower, because we have to also pay the price for the sort by the clustered index
• We have a nearly linear scale

Summary

This article introduced the design pattern for non- und functional partitioned tables. We have seen how we can apply these techniques in SSIS, what the advantages and disadvantages are and how they perform.

But what if loading one month of data is not fast enough? I will take up this question in my next post where we will have a look on hash partitioning and how this technique can help us for further scale.

References

• Design SSIS for Performance and Scale
• Design SSIS for Performance and Scale – Baseline tests
• SSIS Performance Tuning – Methodology and general approach
• SSIS Performance Tuning – Monitoring & Data Collection

• SSIS Design for Performance & Scale: A follow up on the parallelism strategies will be published very soon.
• Parallel Data Warehouse – POC lessons learned: The 3rd part of this series is still missing, but I promise to deliver very soon.

What’s next? I’m currently do some testing and investigation in new technology trends like Big Data and Windows Azure.

It think there is a lot of confusion around Big Data out there. This is now a hype topic like SOA has been. But my hope is, and I’m convinced, that it will have a happy end. If you tell people right now you are a Big Data architect they will probably say “Cool”. But don’t mention you are a SOA guy, I guess they will start laughing “poor boy”.

The point with Big Data is, that everybody talks about, but I don’t see any projects around Big Data technologies like Hadoop, Hive, Pig, etc. Currently Big Data is often used as synonym of handling large amount of data. But it’s much more then a data size topic. So I will take up this trend and give some insights how it could fit to classical data warehouse architectures. I think it’s time that these technologies get closer together.