[Music] in this video we'll discuss some ways to identify import set issues and what you can do about them there are times when things don't always go as planned the first place to look when we run into problems with imports is the system log if the system log doesn't have anything useful you might want to try adding a gs info statement or two to your transform script to ensure it's running as expected then check the logs again a good practice is to test your import set as you build it when you've defined the data source try importing 20 test records check the progress page to ensure 20 records were inserted if you get errors or ignored records check the source data this is also a good time to click the link loaded data to inspect a few import set records to see that the data was imported and the fields are the right type and size i've seen situations where 1000 rows were imported and i ended up with fields named a b c and the values were the field names this told me i was importing a spreadsheet and i had the wrong hetero identified all right what happens if your transform just takes a long time that's when it's best to start asking yourself some of these questions how many records are being imported is this a full import or incremental are business rules running if so what are those business rules doing or workflows or flows or you get the idea how many of those are doing glide record lookups do you understand that do the business rules workflows etc need to run on each record and what happens if you don't perhaps think about this could some of that logic be done using an oncomplete script at the end of the transform let's talk about coalesce fields are those coalesce fields indexed best practice is to do the index after you've verified all the fields being coalesced are correct because frequent changes to indexing can also cause performance issues when in doubt about coalescing indexing and optimization reach out to customer support all alright another question are other imports running at the same time particularly on that same table is a staggered approach possible for example import users at midnight and devices at 4 am to stop an import from running navigate to system import sets advanced progress opening the progress worker to cancel and click cancel job on the related links if the target data has not changed after the import is run check under system import sets advanced progress check the ratio of total count to ignored count and look at ignored records because they indicate records that did not need to be imported because they did not change here's another tip check to ensure that the import set deleter job is running you see the system doesn't need all import set tables indefinitely they're generally only used for your import and then never again each time you run an import it creates a new import set record and the related import set rose the system cleans these out periodically for performance reasons it never hurts to check that it's still running by going to the import sets list and checking how old is the latest import set generally it's 7 days if you see import set records older than that you may have an issue check the scheduled job try running it manually and if it doesn't look right contact customer support when importing a large data set takes a very long time here's a couple of thoughts according to the documentation try breaking the data into smaller chunks instead of 1 million records at once try 10 sets of 100 000 records it's actually faster but try to avoid importing more than 100 000 records at a time if you can second idea try using a data stream if you can this is covered in our integration hub section but it's worth mentioning here since the data source can support a data stream these are great for handling large data sets especially when pagination is required and finally here's a tip that comes in handy when developing your transform map if you need to rerun a transform locate the import set record and change the state to load it by clicking the reprocess ui action then click transform to run it again this saves time since you don't need to actually import all those records again they're already loaded in the import set table now if your import set doesn't go as expected you'll be armed with these handy tips to diagnose and remedy the situation