Build big data pipelines with Azure Data Lake and Azure Data Factory | Azure Blog | Microsoft Azure (2023)

This week, MicrosoftannouncedA public preview of the new and improved Azure Data Lake that makes processing and analyzing big data easier and more affordable. Enhanced Azure Data Lake includes Azure Data Lake Store, Azure Data Lake Analytics, and Azure HDInsight.

Azure Data Lake Store provides a single repository where you can easily ingest data of any size, type, and velocity without forcing changes to your application as the data scales. Azure Data Lake Analytics is a new service built on Apache YARN and includes U-SQL, a language that combines the power of SQL with the expressive power of user code. The service scales dynamically, allowing you to analyze all types of data with enterprise-grade security through Azure Active Directory, allowing you to focus on your business goals.

In the first week of October we haveannouncedWith our existing Azure HDInsight support, you'll be able to build and operationalize big data pipelines (also known as workflows) using Azure Data Lake and Azure Data Factory. Today we publish thempublic reviewof these newly added features. The integration of Azure Data Lake and Azure Data Factory enables you to:

Easily move data to Azure Data Lake Store

Starting today, Azure Data Factory supports moving data from the following sources to Azure Data Lake Store:

• Azure stain
• Azure SQL Database
• Azure chair
• Local SQL Server database
• Azure DocumentDB
• Azure SQL DW
• Local file system
• Local Oracle database
• Local MYSQL database
• Local DB2 database
• Local Teradata database
• Local Sybase database
• PostgreSQL local database
• Local HDFS
• Generic OData (coming soon!)
• Generic ODBC (coming soon!)

You can also move data from Azure Data Lake Store to various sinks such as Azure Blob, Azure SQL Database, local file system and more. Complete the following steps to move data from Azure Blob Storage to Azure Data Lake Store.

Note: You need a valid Azure Data Lake Store account before you can perform the following steps. ClickHereto create a new account if you don't already have one.

Create an Azure data factory

Sign in to the Azure portal and go to Azure Data Factory. Enter a name, select a subscription, a resource group name, and a region name. Let's call it AzureDataLakeStoreAnalyticsSample.

Once built, navigate to your data factory and click Build and Deploy.

Create ADF related services

Create a linked Azure Storage service: This is the Azure Blob storage (source) you want to move data from.

Click New data storage -> Azure Storage. Enter values ​​forIparametre i click Deploy.

Create a linked Azure Data Lake Store service: This is the Azure Data Lake Storage (sink, also called the destination) where you want to move the data.

Click New data store -> Azure Data Lake Store.

Enter the required parameters for the service associated with Azure Data Lake Store

DataLakeUri: created in the step above or using an existing one. For example,

If your Azure Data Lake Store account is in a different subscription and resource group name than your data factory, you must also complete the following parameters:

• Account label
• Subscription ID
• resource group name

Click Deploy. This should create a linked Azure Data Lake Store service.

Note: You must delete the lines that say "Optional" in the JSON if you do not provide values ​​for them before clicking Deploy.

Create ADF records

Create a native Azure Blob Storage dataset:

Click Novi zapis -> Azure Blob Storage.

This will display an Azure Blob Storage record template where you can enter any values. As an example, see the Azure Blob Storage dataset below. For simplicity, we do not use the "partitioned by" clause for time-based partitions and use a static map. The following log shows that the copied data (SearchLog.tsv) is located in the rawdatasample/data/ folder in Azure storage.

{ „name“: „RawBlobDemoTable“, „properties“: { „published“: false, „type“: „AzureBlob“, „linkedServiceName“: „StorageLinkedService“, „typeProperties“: { „fileName“: „SearchLog.tsv“ , „folderPath“: „rawdatasample/data/“, „format“: { „type“: „TextFormat“, „rowDelimiter“: „n“, „columnDelimiter“: „t“ } }, „availability“: { „frequenz ": "Dan", "interval": 1, "style": "StartOfInterval" }, "external": true, "policy": { "validation": { "minimumSizeMB": 0,00001 } } }}

Create the target Azure Data Lake Store dataset:

Click New Record -> Azure Data Lake Store.

This will display the Azure Data Lake Store record template where you can enter any values. For an example, see the following Azure Data Lake Store dataset. For simplicity, we do not use the "partitioned by" clause for time-based partitions and use a static map. The following record indicates that the data has been copied to the datalake/input/ folder in the data lake.

{ "name": "DataLakeTable", "properties": { "published": false, "type": "AzureDataLakeStore", "linkedServiceName": "AzureDataLakeStoreLinkedService", "typeProperties": { "folderPath": "datalake/input/ ", "fileName": "SearchLog.tsv", "format": { "type": "TextFormat", "rowDelimiter": "n", "columnDelimiter": "t" } }, "availability": { "frequenz ": "Oznaka", "Interval": 1 } }}

Build ADF pipelines

ADF Copy Pipeline: This pipeline copies data from Azure Blob Storage to Azure Data Lake.

Click New Pipeline. A sample pipeline template opens. Example: The following pipeline copies data from Azure Blob Storage to Azure Data Lake (sample datasets created above).

Pipeline-Definition:

{ „name“: „EgressBlobToDataLakePipeline“, „properties“: { „description“: „Egress-Daten vom Blob zum Azure Data Lake“, „activities“: [ { „type“: „Copy“, „typeProperties“: { „ izvor ": { "type": "BlobSource", "treatEmptyAsNull": true }, "sink": { "type": "AzureDataLakeStoreSink", "writeBatchSize": 0, "writeBatchTimeout": "00:00:00" } } , „inputs“: [ { „name“: „RawBlobDemoTable“ } ], „outputs“: [ { „name“: „DataLakeTable“ } ], „policy“: { „timeout“: „10:00:00“ , „concurrency“: 1, „executionPriorityOrder“: „NewestFirst“, „retry“: 1 }, „scheduler“: { „frequenz“: „Day“, „interval“: 1 }, „name“: „EgressDataLake“, „ Beschreibung": "Daten vom Blob in den Azure Data Lake verschieben" } ], "start": "2015-08-08T00:00:00Z", "end": "2015-08-08T01:00:00Z", "isPaused": FALSCH }}

Follow ADF pipelines

The ADF copy pipeline created above starts executing because the records have a daily frequency and the start and end are set to 08/08/2015. in the pipeline definition. So the pipelines run only that day and are copied only once. ClickHereto learn more about ADF pipeline planning.

Go to the ADF diagram view to see your data factory pipeline. You can see Azure Blob Storage and Azure Data Lake Store datasets along with the pipeline for moving data from Blob Storage to Azure Data Lake Store.

In the graph view, click DataLakeTable to see the corresponding activity executions and their status.

You can see that the copy activity in EgressBlobToDataLakePipeline in ADF (see screenshot above) succeeded and 3.08 KB of data was copied from Azure Blob Storage to Azure Data Lake Store. You can also log in to the Microsoft Azure portal and use the Azure Data Lake Data Explorer to visualize the data copied to the Azure Data Lake Store.

ClickHereto learn more about data movement activities in Azure Data Factory. See the detailed documentation on using the AzureDataLakeStore connector in ADFHere.

Build E2E Big Data ADF pipelines that run U-SQL scripts as a processing step in Azure Data Lake Analytics

A very common use case across multiple industries (retail, finance, gaming) is log processing.

Note: You need a valid Azure Data Lake Analytics account before you can perform the following steps. ClickHereto create a new account if you don't already have one.

In this scenario, you create an ADF pipeline that uses the logs copied to the Azure Data Lake Store account in the previous step and processes the logs by running a U-SQL script in Azure Data Lake Analytics as one of the processing steps. The U-SQL script calculates events by region that can be used by downstream processes.

We will reuse the data factory (AzureDataLakeStoreAnalyticsSample) created in the above scenario to copy data from Azure Blob Storage to Azure Data Lake Store.

Create ADF related services

Create a service connected to Azure Data Lake Analytics. This is the Azure Data Lake Analytics account that runs the U-SQL scripts to process the logs.

Click New Computing -> Azure Data Lake Analytics.

Enter the required parameters for the associated Azure Data Lake Analytics service

• Account Name: Created in the step above or using an existing account
• Authorization: To complete this parameter, click Authorize. This will open a popup asking you to enter your credentials.

Enter optional parameters if your Azure Data Lake Analytics account is in a different subscription and resource group name.

• Subscription ID
• resource group name

Click Deploy. This should create a connected Azure Data Lake Analytics service.

Note: You must delete the lines that say "Optional" in the JSON if you do not provide values ​​for them before clicking Deploy.

Create a linked Azure Data Lake Store service: This is the Azure Data Lake Storage (sink, also called the destination) where you want to move the data.

Note: If you are running this scenario as a continuation of the copy scenario above, you have already created this linked service.

Click New data store -> Azure Data Lake Store.

Enter the required parameters for the service associated with Azure Data Lake Store

DataLakeUri: created in the step above or using an existing one), example:

If your Azure Data Lake Store account is in a different subscription and resource group name than your data factory, you must also complete the following parameters:

• Account label
• Subscription ID
• resource group name

Click Deploy. This should create a linked Azure Data Lake Store service.

Note: You must delete the lines that say "Optional" in the JSON if you do not provide values ​​for them before clicking Deploy.

Create ADF records

Create the original Azure Data Lake Store dataset:

Note: If you are running this scenario as a follow-up to the copy scenario above, you have already created this record.

Click New Record -> Azure Data Lake Store.

This brings up the Azure Data Lake Store record template. You can enter any values.

For example, consider the following Azure Data Lake Store dataset. For simplicity, we do not use the "partitioned by" clause for time-based partitions and use a static map. The following record indicates that the data has been copied to the datalake/input/ folder in the data lake.

{ "name": "DataLakeTable", "properties": { "published": false, "type": "AzureDataLakeStore", "linkedServiceName": "AzureDataLakeStoreLinkedService", "typeProperties": { "folderPath": "datalake/input/ ", "fileName": "SearchLog.tsv", "format": { "type": "TextFormat", "rowDelimiter": "n", "columnDelimiter": "t" } }, "availability": { "frequenz ": "Oznaka", "Interval": 1 } }}

Create an Azure Data Lake Store target record.

Click New Record -> Azure Data Lake Store.

Example: See the EventsByEnGbRegionTable dataset definition below. The data corresponding to this record is created after running the AzureDataLakeAnalytics U-SQL script to get all events for locale 'en-gb' and date < '02/19/2012'.

{ "name": "EventsByEnGbRegionTable", "properties": { "published": false, "type": "AzureDataLakeStore", "linkedServiceName": "AzureDataLakeStoreLinkedService", "typeProperties": { "folderPath": "datalake/output/ " }, "dostupnost": { "Frequenz": "Oznaka", "Interval": 1 } }}

Build ADF pipelines

Create an ADF AzureDataLakeAnalytics pipeline: This pipeline runs a U-SQL activity for processing.

Click New Pipeline. A sample pipeline template opens.

You can also click Add Activity after clicking New Pipeline and add the DataLakeAnalytics SQL Activity Template.

For example, the following pipeline runs an ADLA U-SQL activity to retrieve all events for locale en-gb and date < 2012-02-19.

Pipeline-Definition:

{ "name": "ComputeEventsByEnGbRegionPipeline", "properties": { "description": "Ovo je cjevovod za izračunavanje događaja za en-gb lokalizaciju i datum prije 02/19/2012.", "activities": [ { " type ": "DataLakeAnalyticsU-SQL", "typeProperties": { "scriptPath": "scriptskonaSearchLogProcessing.txt", "scriptLinkedService": "StorageLinkedService", "degreeOfParallelism": 3, "priority": 100, "parameters": { " in ": "/datalake/input/SearchLog.tsv", "out": "/datalake/output/Result.tsv" } }, "inputs": [ { "name": "DataLakeTable" } ], "outputs " : [ { "name": "EventsByEnGbRegionTable" } ], "policy": { "timeout": "06:00:00", "concurrency": 1, "executionPriorityOrder": "NewestFirst", "retry": 1 } , "scheduler": { "frequency": "Dan", "interval": 1 }, "name": "EventsByRegion", "linkedServiceName": "AzureDataLakeAnalyticsLinkedService" } ], "start": "2015-08-08T00 : 00 :00Z", "end": "2015-08-08T01:00:00Z", "isPaused": false }}

The U-SQL script executed by the above pipeline is located in the scripts/cones folder in the Azure Blob Storage account corresponding to the deployed StorageLinkedService service.

SearchLogProcessing.txt-Skriptdefinition:

@searchlog = EXTRACT UserId int, start datetime, string region, string query, duration int?, string urls, string ClickedUrls FROM @in USING Extractors.Tsv(nullEscape:"#NULL#");@rs1 = SELECT Start, Region, Duration FROM @searchlogWHERE Region == "en-gb";@rs1 = SELECT Start, Region, Duration FROM @rs1 WHERE Start <= DateTime.Parse("2012/02/19");OUTPUT @rs1 TO @out USING output programs .Tsv(quote:false, dateTimeFormat:null);

The values ​​for the @in and @out parameters in the U-SQL script above are dynamically passed to ADF using the Parameters section. See the Parameters in Pipeline Definition section above.

You can specify other properties, eg You can also specify "degreeOfParalleism", "priority", etc. in your pipeline definition for jobs running in Azure Data Lake Analytics.

Follow ADF pipelines

The above ADF copy pipeline starts executing because the records have a daily frequency and the start and end are set to 08/08/2015. in the pipeline definition. Therefore, the pipelines run only on that day and run the U-SQL script once. ClickHereto learn more about ADF pipeline planning.

Go to the ADF diagram view to see your data factory pipeline. You will see two pipelines and their corresponding records ie. EgressBlobToDataLakePipeline (copy data from Azure Blob Storage to Azure Data Lake Store) and ComputeEventsByEnGbRegionPipeline (get all events for locale 'en-gb' and date < '02/19/2012').

In the graph view, click EventsByEnGbRegionTable to see the corresponding activity executions and their status.

You can see that the U-SQL activity in ComputeEventsByEnGbRegionPipeline in ADF has run successfully and created a Results.tsv file (/datalake/output/Result.tsv) in your AzureDataLakeStore account. Result.tsv contains all events for locale "en-gb" and date < "02/19/2012". You can log in to the Microsoft Azure portal and use the Azure Data Lake Data Explorer to view the Result.tsv file generated as part of the above processing step in the Azure Data Lake Store.

For detailed documentation on the AzureDataLakeAnalytics SQL activity, see Azure Data FactoryHere.

In short, by following the steps above, you have been able to create E2E big data pipelines using Azure Data Factory, allowing you to move data to Azure Data Lake Store. In addition, you can run U-SQL scripts in Azure Data Lake Analytics as one of the processing steps and dynamically scale according to your needs.

We will continue to invest in solutions that allow us to operationalize the processing of big data and analytical work processes. ClickHereLearn more about Microsoft Azure Data Lake from the Microsoft Cloud Platform team. If you want to try Azure Data Factory, visit usHereand start building pipelines easily and quickly using Data Factory. If you have feature requests or would like to provide feedback on Data Factory, visitAzure Data Factory-Forum.