Re-create data from or to MongoDB Atlas using Azure Information Factory or Synapse Analytics

  • Article
  • 5 minutes to read

APPLIES TO: Azure Data Factory Azure Synapse Analytics

This article outlines how to use the Re-create Activity in an Azure Data Manufactory or Synapse Analytics pipeline to copy information from and to a MongoDB Atlas database. It builds on the copy activity overview article that presents a general overview of copy activity.

Supported capabilities

You can re-create information from MongoDB Atlas database to any supported sink data shop, or copy data from any supported source data shop to MongoDB Atlas database. For a list of data stores that are supported as sources/sinks by the re-create activity, see the Supported data stores table.

Specifically, this MongoDB Atlas connector supports versions up to four.ii.

Prerequisites

If y'all use Azure Integration Runtime for copy, make sure you add the effective region's Azure Integration Runtime IPs to the MongoDB Atlas IP Access Listing.

Getting started

To perform the Re-create activity with a pipeline, you can utilize i of the following tools or SDKs:

  • The Re-create Data tool
  • The Azure portal
  • The .NET SDK
  • The Python SDK
  • Azure PowerShell
  • The Rest API
  • The Azure Resources Manager template

Create a linked service to MongoDB Atlas using UI

Use the following steps to create a linked service to MongoDB Atlas in the Azure portal UI.

  1. Browse to the Manage tab in your Azure Information Factory or Synapse workspace and select Linked Services, and then click New:

    • Azure Information Manufactory
    • Azure Synapse

    Create a new linked service with Azure Data Factory UI.

  2. Search for MongoDB and select the MongoDB Atlas connector.

    Select the MongoDB Atlas connector.

  3. Configure the service details, examination the connexion, and create the new linked service.

    Configure a linked service to MongoDB Atlas.

Connector configuration details

The following sections provide details most properties that are used to ascertain Data Manufacturing plant entities specific to MongoDB Atlas connector.

Linked service backdrop

The post-obit properties are supported for MongoDB Atlas linked service:

Property Clarification Required
type The type property must be ready to: MongoDbAtlas Yeah
connectionString Specify the MongoDB Atlas connection string e.thousand. mongodb+srv://<username>:<password>@<clustername>.<randomString>.<hostName>/<dbname>?<otherProperties>.

You lot tin also put a connection string in Azure Primal Vault. Refer to Shop credentials in Azure Key Vault with more details.

 Yes
database Name of the database that y'all want to access. Yes
connectVia The Integration Runtime to be used to connect to the data shop. Larn more from Prerequisites department. If not specified, information technology uses the default Azure Integration Runtime. No

Example: 

              {     "proper noun": "MongoDbAtlasLinkedService",     "backdrop": {         "blazon": "MongoDbAtlas",         "typeProperties": {             "connectionString": "mongodb+srv://<username>:<countersign>@<clustername>.<randomString>.<hostName>/<dbname>?<otherProperties>",             "database": "myDatabase"         },         "connectVia": {             "referenceName": "<name of Integration Runtime>",             "type": "IntegrationRuntimeReference"         }     } }

Dataset properties

For a full list of sections and properties that are available for defining datasets, meet Datasets and linked services. The following properties are supported for MongoDB Atlas dataset:

Property Description Required
type The blazon property of the dataset must exist set to: MongoDbAtlasCollection Yep
collectionName Proper noun of the collection in MongoDB Atlas database. Yes

Example: 

              {     "name": "MongoDbAtlasDataset",     "properties": {         "type": "MongoDbAtlasCollection",         "typeProperties": {             "collectionName": "<Collection proper noun>"         },         "schema": [],         "linkedServiceName": {             "referenceName": "<MongoDB Atlas linked service proper name>",             "type": "LinkedServiceReference"         }     } }

Copy activeness backdrop

For a full list of sections and properties available for defining activities, see the Pipelines article. This section provides a list of backdrop supported by MongoDB Atlas source and sink.

MongoDB Atlas as source

The post-obit properties are supported in the copy activity source department:

Belongings Description Required
blazon The blazon property of the copy action source must be set to: MongoDbAtlasSource Yes
filter Specifies selection filter using query operators. To return all documents in a drove, omit this parameter or pass an empty document ({}). No
cursorMethods.project Specifies the fields to return in the documents for project. To return all fields in the matching documents, omit this parameter. No
cursorMethods.sort Specifies the order in which the query returns matching documents. Refer to cursor.sort(). No
cursorMethods.limit Specifies the maximum number of documents the server returns. Refer to cursor.limit(). No
cursorMethods.skip Specifies the number of documents to skip and from where MongoDB Atlas begins to render results. Refer to cursor.skip(). No
batchSize Specifies the number of documents to return in each batch of the response from MongoDB Atlas example. In most cases, modifying the batch size volition non bear on the user or the application. Creation DB limits each batch cannot exceed 40MB in size, which is the sum of the batchSize number of documents' size, so decrease this value if your document size being big. No
(the default is 100)

Tip

The service supports consuming BSON document in Strict manner. Make certain your filter query is in Strict mode instead of Shell mode. More clarification can exist found at MongoDB transmission.

Case: 

              "activities":[     {         "name": "CopyFromMongoDbAtlas",         "blazon": "Copy",         "inputs": [             {                 "referenceName": "<MongoDB Atlas input dataset name>",                 "blazon": "DatasetReference"             }         ],         "outputs": [             {                 "referenceName": "<output dataset name>",                 "type": "DatasetReference"             }         ],         "typeProperties": {             "source": {                 "type": "MongoDbAtlasSource",                 "filter": "{datetimeData: {$gte: ISODate(\"2018-12-11T00:00:00.000Z\"),$lt: ISODate(\"2018-12-12T00:00:00.000Z\")}, _id: ObjectId(\"5acd7c3d0000000000000000\") }",                 "cursorMethods": {                     "project": "{ _id : 1, proper name : i, age: ane, datetimeData: 1 }",                     "sort": "{ historic period : 1 }",                     "skip": 3,                     "limit": 3                 }             },             "sink": {                 "type": "<sink type>"             }         }     } ]

MongoDB Atlas as sink

The following properties are supported in the Copy Action sink section:

Property Description Required
type The type property of the Copy Activity sink must be set to MongoDbAtlasSink. Yes
writeBehavior Describes how to write data to MongoDB Atlas. Immune values: insert and upsert.

The behavior of upsert is to supplant the certificate if a document with the same _id already exists; otherwise, insert the certificate.

Annotation: The service automatically generates an _id for a certificate if an _id isn't specified either in the original document or by column mapping. This means that y'all must ensure that, for upsert to work as expected, your document has an ID.

 No
(the default is insert)
writeBatchSize The writeBatchSize property controls the size of documents to write in each batch. You can try increasing the value for writeBatchSize to improve performance and decreasing the value if your document size beingness large. No
(the default is 10,000)
writeBatchTimeout The wait time for the batch insert operation to terminate before it times out. The immune value is timespan. No
(the default is 00:30:00 - 30 minutes)

Tip

To import JSON documents as-is, refer to Import or consign JSON documents section; to copy from tabular-shaped data, refer to Schema mapping.

Instance 

              "activities":[     {         "name": "CopyToMongoDBAtlas",         "type": "Copy",         "inputs": [             {                 "referenceName": "<input dataset proper noun>",                 "type": "DatasetReference"             }         ],         "outputs": [             {                 "referenceName": "<Document DB output dataset name>",                 "type": "DatasetReference"             }         ],         "typeProperties": {             "source": {                 "type": "<source type>"             },             "sink": {                 "type": "MongoDbAtlasSink",                 "writeBehavior": "upsert"             }         }     } ]

Import and Export JSON documents

Y'all can use this MongoDB Atlas connector to easily:

  • Copy documents betwixt ii MongoDB Atlas collections as-is.
  • Import JSON documents from various sources to MongoDB Atlas, including from Azure Cosmos DB, Azure Blob storage, Azure Data Lake Shop, and other supported file-based stores.
  • Consign JSON documents from a MongoDB Atlas drove to diverse file-based stores.

To achieve such schema-agnostic re-create, skip the "structure" (as well called schema) section in dataset and schema mapping in copy activity.

Schema mapping

To re-create data from MongoDB Atlas to tabular sink or reversed, refer to schema mapping.

Adjacent steps

For a list of data stores supported equally sources and sinks by the re-create activity, encounter supported data stores.