[Music] in this video we'll cover the concept of concurrent import sets if we've got more than a hundred thousand records to import we might consider using a concurrent import set now in this context this doesn't mean running multiple imports at the same time but a single data source that processes its import sets and transforms them concurrently spreading the work across multiple nodes in your instance cluster thus reducing overall processing time that being said concurrent import sets actually add processing and monitoring overhead to the system but overall they can reduce the time it takes to import the data so we recommend using them with large data sets only in the previous video we saw how to schedule an import by going to system import sets administration scheduled imports let's go through the process of taking an existing data source and scheduling a concurrent import for it we'll create the new schedule record and give it a name point it to our data source and for this example i'll use my account in the run as field however remember our best practice from last video we should really be using the dedicated integration account we're also going to tell it to run once a day at 1 am when we check concurrent import it tells the system to create multiple import sets the number of import sets is based on the number of nodes making up your instance typically this is two times the number of nodes for example if we have two nodes the system creates four import sets the maximum number of import sets is controlled by the system properties seen here so even if we have 10 nodes in our instance cluster we won't get more than 10 import sets if the property is 10. for this example we only have 3000 records for the demonstration purposes so we'll change the partition method to custom size and the partition size to 1000 records per import set we could also create a custom script to return a key that ensures all records with the same key end up in the same import set we don't need any script run before or after the import so we'll leave these two unchecked we save the record and click execute now behind the scenes there's a job called the import set transformer that runs every minute on every node it pulls the concurrent import set jobs queue to pick an import set and transform it these jobs run concurrently so if our data needs to be processed in a specific order like creating users before groups then concurrent jobs won't be a good fit to monitor the progress of concurrent import set we can go to system import sets advanced concurrent import sets and find the latest one and look at the related records directly from the list or by opening the record if we want to reprocess the entire set of import set records we can do it from the parent concurrent import set or reprocess an individual import set from its record with the same reprocess ui action in the related links the import set records are pretty much the same as any other import set except for the concurrent import set field in an earlier episode on transform mapping fields we talked about coalesce fields in short fields identified as coalesce tell the system when to create a new record and when to update an existing one so what happens when concurrent import sets are running and more than one identify a record to create thankfully concurrent import sets use a locking mechanism so the first insert locks the target table to prevent another import set from inserting the same record hopefully this example provides enough information to understand when to use concurrent import sets and how to configure and monitor them if you still have questions check the description of this video for links or add your comments