logo

NJP

#7 Data source script

Import · Jul 21, 2020 · video

well hi everyone so in this video we are going to create our data source which is actually going to call our flow designer action okay so we are going to get the data for example the historical data and also the coin details so we are going to write a script because the type of the data source is going to be load by script so let's go here in this sheet to data source load by script i'm going to write our script here so same thing that we have done for the previous one i'm gonna have all the kind of visual representation the logic and then we are going to write here our script so the first thing we are going to do is we are going to take the core snippet that we have in flow designer okay let's go to flow designer this is the action that we have created and if you click on the three dots that you have here click on code snippet this is the code that we have to use so we don't need the entire code all we need is this part from what you have in the try uh let's say variable here so you have to use this one okay so i have all already here all the code so i'm going to just take this entire code snippet let's say i'm going to put it here and one more thing make sure that you have your action published otherwise you cannot use it in your data source so here uh what we have we we have for example the input variable and it is empty for now inputs coin id we have nothing and we are calling the flow designer action with our input and then we get some outputs okay we have coin object and then coin array a b c d until h okay so we know that we have to go for we have to use this uh action eight time okay we cannot just uh have call it once and it is done we we have to call eight times uh we have to use this code a time so how we are going to do that so we can have a for loop okay each time when we are running the for loop we will have a different input we're going to have for the first time one second time 13 third time another number so these are the coin id okay so before creating our for loop the first thing what i'm going to do i'm going to define all the all my coins here so i have it here sorry so i have all bitcoin equal to 1 well dash equal to 13 and then so on so i'm going to take this code copy and then put it in my code here okay so i'm defining all my coins then i'm going to create an array which is going to be coin sorry and it will have one error inside and it will contain all the coins so here i have the visual representation so this is the coin array for example and uh at the position coins 0 0 i will have my bitcoin and then so on zero one dash co2 decree and then so on so i'm gonna just take the code that we have here let me check if i put it okay let's go and then take this code and then paste it here okay so i have some sort of problem now let's see what's going on here why i don't have the entire all right yes now i have it okay so [Music] now i'm gonna have my for loop so for bar a equal to zero a smaller than eight because we have a coins a plus plus okay i'm gonna have the entire code inside okay so i'm gonna run eight to eight loops and here i'm gonna have your coins brackets brackets and you know if i do zero zero i'm gonna get bitcoin and zero one i'm gonna get dash so the first value that i'm going to have in the first bracket it's going to be always zero for all coins we have zero the one thing which is changing is the second what we have in the second bracket so if you want you can look one more time this uh table and you can see zero zero it's bitcoin zero one it is dash okay and then zero two is degree so instead of so here i'm gonna have the variable a because we know a is increasing from zero to seven so we can have all the coins like this and we will send this as input it will call the flow designer action and then we will get the outputs for that coin one thing that i want to mention here you can see here the comment here is integer int right if you go to flow designer open here click on code snippet if you haven't i haven't i've just copied the code and put it there but the thing in when you write when you're creating your data source when you put this code into your data source and if you put a value like let's say i'm putting one like the number it is not working it is not calling and you have to put like this like into quotation for some reason it should be i guess a string um if you put just a number it should work but it is not working so i have to like change it as a string here so i'm to type another code here syntax which is actually going to change the value into a string so i'm going to do like this coins 0 a dot 2 string same parenthesis it is actually going to change the value that we are going to have here as a string and now the code in your data source it will not fail you can test it if you want okay that's something that i would like to mention here so now we um we're gonna have all the data here for all our coins but now we have to know how we are going to take the data from these objects and put it uh into our staging table for that the first thing that i'm going to do i'm going to create a historical price array which will contains all my arrays a b c d everything okay so i'm gonna take the code that i have here first okay copy and then put it here then i'm going to create this historical array historical price array so which i have it here historical price coiner a b c d and then so on i'm going to just copy everything and put it here and it should come here because you should put it here not in the beginning because these values while coins are a b it's only defined at this point it's not defined here so this code you should come after so we are defining like we are putting all these uh coins are a b c everything together into like one big array okay which contains all these arrays now let's go here i'm going to write a for loop okay let's imagine that we are retrieving data for bitcoin for one coin and then we'll see how we are going to do for other coins so let's say this is bitcoin okay and this value should be 119 okay the length is 120. so i'm going to have the first for loop which is going to run from for bar equal to for bar i equal to 0 1 2 3 4 until 7. and then for r i equal to 0 i equal to 0 i'm going to have another loop which is going to be j equal to 0 and then it is going until 244 because the length that i have is 245 okay we know already the length of our arrays okay we are not defining it here we know those values we have it here okay that's that's one thing so and our area is it's not empty we have time and price inside okay time and price for each cell they're having their they have time and price so by writing two for loop we can exactly access this part of the the cell okay this entire big historical price area we can we're going to use it uh enough you know for loops to access the this particular memory to take the time time and price okay it is exactly the same logic that we have used before in our rest request i mean in the the when we were passing a response uh to access those values and then get those values in from the history array so the same thing that we are doing here okay so i'm going here [Music] and then i'm going to write my first for loop var i'm going to have i equal to zero i smaller than what historical price dot length okay i want to run the loop for this length for eight time right because we have eight arrays then i'm going to do i okay here here i plus plus okay okay what's going on up then i'm going to have my second loop it's going to be 4 1 4 sorry bar j equal to zero j smaller than what i should have the length of my coins array a so the first thing to do is to is to know how we can access this particular array from historical price so all i need to do is to type historical price brackets and brackets okay we know if we if we are doing let's go back here 0 0 is going to be coin error a 1 0 is going to be corner b so i'm going to have here the second value is going to be always zero so i'm going to put that one here but the first value can be 0 1 2 3 4 5 6 7 okay which is i in our case here so that's it so we do that and then what we want is the length for that specific all right okay so and then i do j plus plus so let me just go here i think i have something wrong here okay this one i'm closing it here so let's make sure that we have proper curly brackets here so this one is this for this one this one is for this one and this one is for this one great so now what we are going to do is we are going to create our objects specifically i'm going to call it map object which is actually going to insert all the values i mean not insert we will prepare like we will collect the object and we will have in a certain fields and then we will insert later into our staging table so let's do that so that's going to be our next part here down below and i'm gonna just copy the entire code that we have here up okay and i'm going to write my code here so i'm creating a map object and inside i'm gonna have everything so [Music] let me just write the first example how we are going to get some data so i'm gonna have you underscore id and then here i'm gonna have coin object okay i'm going i'm going to use this one current object dot id so i can get the id so i can do the same in order to get all other information so if you look at carefully what we have here in this object we have id name slug website url all this kind of information so i have it already here so i'm going to just take that part and then put it here up okay it is okay i was putting semi column it's not semicolon it's just a comma so the next thing these are the coin details so we are using this object okay then the main thing that we want is the uh is the time and the price okay so let's get that let's get that as well okay so let me just finish with this one let's arrange everything okay now okay okay is it some way i'm sorry for this okay now we are going to do [Music] take the data for we are going to have our price and and the time so i'm going to write the same way you underscore price this time we are going to use our historical price array in our axis for example coinary a so i'm going to type here historical price to twice i'm gonna have what i should for the first loop i should have um coins area a right so i'm going to have here i and then 0 in this way i can get a right then i'm going to have another brackets so once i access a coins are a the length is 245 and we will have j will be equal to zero but it will go from zero to one two three four five until 244 so i'm gonna have here j so this is how i'm gonna i'm accessing basically the historical price that specific cell that we actually saw with this one okay and then it is not done so let's go back here let's finish so that price okay and then come on so we are going to do the same thing to get the timestamp so let's go here i have it here i'm going to just take it from here up and then put the timestamp here so we have created our object which will contain all the data that we need okay here we go okay i have a problem okay so we have our map object we have all the uh let's say all the information that we need and we are storing it okay right we are creating an object which has all the values and then we have to insert this object i mean create fields in our stage in table and which will have all these values for that i'm going to use this specific syntax which is imports under import underscore set underscore table dot insert and then i'm gonna have here the map object so i'm gonna just take this code and put it exactly right here after the map object so in this way once we have on that specific cell okay we get the data okay we have our map object immediately we do an insert so for the first loop we are going to do 244 45 insert for the second loop same thing third loop same thing we will go until the end which is the coin array h loop and we will do for 120 times in total we are going to get for a hundred and tell sorry 1825 elements 1025 inserts we're gonna have in our staging table okay so this is this is good for bitcoin okay but it is not good for any other coin if you take uh bitcoin sv we don't need as many arrays actually so it's going to stop at a certain moment we need actually uh how many uh i guess for these coins for the last three three uh coins we just need five arrays okay so we don't need eight and same way for other coins like a chain link we just need six only for bitcoin and dash and decreed we need eight arrays so now let's see how we can modify a little bit our code in order to just use the arrays that we need okay now we don't need all arrays so let's go a little bit here and i'm going to show you how what we are going to do the first thing what we know if you look at the code uh carefully historical price.length it is always the same if you take if you calculate i mean if you take the value if you just look for the value what is the length historical price it has eight elements so it's going to be always eight in our loop okay so that's what i have here historical price length eight it is fixed it is there right and here i have historical price length expected okay what we expect um to have as length because depending on the current that we have the length is not the same so i'm going to put the length here so if it is bitcoin i want eight if it is dash i want eight if it is decreed i want eight but if it is below coin i don't need eight arrays i need only six okay so i'm gonna have six as well for chain link if it is bitcoin cash or sv or cosmos it is going to be 5 right 5 and then 5 here so another question how i can move from this value to this value depending on the coin that i have okay so if i have bitcoin cash i should not have this value for historical price length but instead i should have this value so here's a simple calculation it's a subtraction okay if you want to move from here to from eight to five all you need to do is to subtract with the three so eight minus three i'm gonna get five so i'm gonna have three here i'm gonna have three here and here i'm gonna have two and i have two and here i'm gonna have zero zero zero so what i'm doing here i'm gonna have a variable called x which will contain these values that i'm going to use in my code so here i'm going to just take historical length which i have already which is in my code okay defined minus that specific variable which will contain all these values so i'm going to have here x that's the variable okay but x were x i should send these values how i can assign these values to to this specific variable x so for that what i'm going to do i'm going to take all these values create another array in my coins array so if you remember in the beginning we have coins array which has only the coins i'm going to have a second one which is going to have these values so i'm going to have here 0 0 0 2 2 and then 3 here 3 3 and then the last one is going to be 3 as well so this is going to be for the endless one so if you want to access this specific value what you should do it's coin 1 0 and then 1 1 1 2 1 3 1 4 and then so on so x will be equal to coins brackets brackets the first value is going to be always one okay the second value is going to change from zero to seven which is actually if you take the loop the value what we what we will have in the variable a right so i'm going to just replace i'm going to put here a that's it so we can use this specific syntax we're going to create this variable bar x equal to coins on a which will have each time the value that we want so if we have as input bitcoin in our loop so we're going to run the loop and let's say um the current id we are sending here one okay and it's going to run it's going to execute okay for bitcoin it's going to get the data and here it will come at this point okay i'm going to have var x equal to coins okay brackets brackets and then and that will be equal to one here i'm gonna have a this so when here a will be equal to zero we have bitcoin and then one zero is the value 0 that's what we have here so x will be equal to 0 right so all i need to do for now if x is equal to 0 i'm gonna just do here minus x okay historical price minus x which is eight minus zero that's what we have here in this uh specific place okay so for bitcoin is going to be zero for dash is going to be zero for degree is going to be zero i'm gonna have eight minus zero i'm gonna get eight but for other coins let's say for cosmos same logic let's say um i mean the end of the loop okay so a will be equal to seven and coins so i will have here uh cosmos as coin is going to execute the action i'm going to get the uh all the outputs then here we have coins one seven okay which is this one okay and for one seven the value that i'm going to have in x is going to be three that's what i have defined okay and uh it is and then if that that is three i'm gonna have historical price dot length minus three which is going to be five so it's going to have only number five as land here okay so this is how we are actually defining the length for for in our loop so before we forget just let's add this these values here so i have it here i guess oh i don't have it anywhere so you have to make sure that you add this array so i'm gonna just add it now okay so brackets the first one is 0 comma 0 0 and then we have 2 2 and then we have 3 three okay so we should have eight one two three four five six seven eight so this is what you should have here so all the values that we have here three times zero two time two three times three that's it so we have our core now so in the next video what we are going to do we are going to just take this code put it in our data source and load our data and see if we are going to have uh all the data into our staging table so see you in the next video thanks for watching bye

View original source

https://www.youtube.com/watch?v=7BXiOb9Jb2Y